feat: support retriable stall with controllable delay

[raj-prince]

Fixes #667

• [x] Tests pass
• [x] Appropriate changes to README are included in PR

[conventional-commit-lint-gcf[bot]]

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot https://conventionalcommits.org/

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.70%. Comparing base (61bce3c) to head (e53b8f4). Report is 32 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #666 +/- ## ========================================== + Coverage 98.69% 98.70% +0.01% ========================================== Files 50 50 Lines 8492 8527 +35 ========================================== + Hits 8381 8417 +36 + Misses 111 110 -1 ``` | [Flag](https://app.codecov.io/gh/googleapis/storage-testbench/pull/666/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=googleapis) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/googleapis/storage-testbench/pull/666/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=googleapis) | `98.70% <100.00%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=googleapis#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.