jd / tenacity

Retrying library for Python
http://tenacity.readthedocs.io
Apache License 2.0
6.82k stars 283 forks source link

Add ability to inspect upcoming sleep in `stop` funcs, and add `stop_before_delay` #423

Closed christek91 closed 11 months ago

christek91 commented 1 year ago

This seeks to solve a specific problem: if using a wait function like wait_exponential or wait_random_exponential, along with stop_after_delay, the overall time spent retrying is >= max_delay.

However, in situations where you have a strict deadline and you can't block for longer than max_delay, it may be preferable to abort 1 retry attempt early so that you don't exceed that deadline.

This PR seeks to achieve that behavior by implementing stop_before_delay, which stops retries if the next attempt would take place after the max_delay time has elapsed, thereby ensuring that max_delay is never surpassed.

Examples of problem using exponential delays, and a 5 second deadline w/ stop_after_delay:

In [5]: @retry(stop=stop_after_delay(5), wait=wait_exponential())
   ...: def stop_after_w_e():
   ...:     print(datetime.now())
   ...:     raise Exception

In [11]: stop_after_w_e()
2023-11-08 23:18:12.259584
2023-11-08 23:18:13.261515
2023-11-08 23:18:15.262064
2023-11-08 23:18:19.263860
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

Note how the execution timestamp is 7 seconds after the initial one, despite the 5 second max_delay.

In [6]: @retry(stop=stop_after_delay(5), wait=wait_random_exponential())
   ...: def stop_after_w_r_e():
   ...:     print(datetime.now())
   ...:     raise Exception
   ...:

In [12]: stop_after_w_r_e()
2023-11-08 23:19:02.682304
2023-11-08 23:19:02.991432
2023-11-08 23:19:04.510066
2023-11-08 23:19:07.398501
2023-11-08 23:19:14.172982
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

Note the same thing w/ random exponential backoff. In this case the final attempt is made 11.49 seconds after the first one, despite a 5 second max_delay. That's over 2x the amount of time spent blocking on retries than our strict deadline allows!

This situation with random exponential backoff is the one that I'm interested in improving upon for my own use case (avoiding thundering herds during connection retries from many parallel clients). This extra time spent waiting after max_delay can be quite long with higher amounts exponential backoff. eg. a supposedly 30s max_delay that actually blocks for over 1 minute can cause some confusion.

With stop_before, the max_delay is never exceeded:

In [12]: @retry(stop=stop_before_delay(5), wait=wait_exponential())
    ...: def stop_before_w_e():
    ...:     print(datetime.now())
    ...:     raise Exception

In [13]: stop_before_w_e()
2023-11-08 23:21:56.437116
2023-11-08 23:21:57.439288
2023-11-08 23:21:59.441351
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

Total elapsed time is < 5 seconds (3 seconds).

In [14]: @retry(stop=stop_before_delay(5), wait=wait_random_exponential())
    ...: def stop_before_w_r_e():
    ...:     print(datetime.now())
    ...:     raise Exception

In [15]: stop_before_w_r_e()
2023-11-08 23:23:13.550395
2023-11-08 23:23:14.344831
2023-11-08 23:23:15.533050
2023-11-08 23:23:17.907447
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

Total elapsed time is < 5 seconds (4.35).