Fix issue 5972

[karolz-ms]

Description

Turns out that we did not wait long enough for the Container graceful stop to occur during restart sequence. This change tweaks the timeout used there.

Honestly I am not exactly sure how Polly calculates the exponential delay periods. Mathematically the original retry strategy settings should have been sufficient, but in reality we have been waiting too short. Polly also lacks the equivalent of MaxElapsedTime present in other retry frameworks, so I have verified experimentally that the current settings give desired max retry time.

Fixes #5972

Checklist

• Is this feature complete?
  • [x] Yes. Ready to ship.
  • [ ] No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • [ ] Yes
  • [x] No (current tests sufficient)
• Did you add public API?
  • [ ] Yes
  • If yes, did you have an API Review for it?
    • [ ] Yes
    • [ ] No
  • Did you add <remarks /> and <code /> elements on your triple slash comments?
    • [ ] Yes
    • [ ] No
  • [x] No
• Does the change make any security assumptions or guarantees?
  • [ ] Yes
  • If yes, have you done a threat model and had a security review?
    • [ ] Yes
    • [ ] No
  • [x] No

• Does the change require an update in our Aspire docs?

  • [ ] Yes
  • Is this introducing a breaking change?
    • [ ] Yes
    • Link to aspire-docs issue (please use this breaking-change template):
    • [ ] No
    • Link to aspire-docs issue (please use this doc-idea template):
  • [x] No

  Microsoft Reviewers: Open in CodeFlow

[davidfowl]

cc @martincostello for polly comment

[martincostello]

For the behaviour you're after (don't exceed this amount of time in total) you'd typically add a timeout policy to the resilience pipeline. Then the cancellation token for that would be passed from the timeout policy to the retry one, and it would observe it and terminate the retries when it's signalled.

[karolz-ms]

Thanks @martincostello I will keep this in mind for the next time when I am working in this part of the codebase.