atlassian-api / atlassian-python-api

Atlassian Python REST API wrapper
https://atlassian-python-api.readthedocs.io
Apache License 2.0
1.29k stars 642 forks source link

[Rest] Support exponential backoff and retry #1339

Closed flichtenheld closed 4 months ago

flichtenheld commented 4 months ago

Uses the urllib3 builtin Retry mechanism. We only support retrying on HTTP return status, not on any of the other retries that that class defines (e.g. connection problems, redirects).

The usual use case for this is to handle 429 responses from the Atlassian Cloud API which has request limits that are easily hit by scripts iterating over objects like Jira issues or Bitbucket pull requests.

Based on earlier work by @jp-harvey.

Replaces #904

flichtenheld commented 4 months ago

The retries are only visible on DEBUG level logging. Example:

2024-02-26 16:25:45,497 DEBUG:curl --silent -X GET -H 'Content-Type: application/json' -H 'Accept: application/json'  'https://api.bitbucket.org/2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch'
2024-02-26 16:25:45,721 DEBUG:https://api.bitbucket.org:443 "GET /2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch HTTP/1.1" 429 46
2024-02-26 16:25:45,721 DEBUG:Incremented Retry for (url='/2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch'): Retry(total=None, connect=None, read=None, redirect=None, status=999)
2024-02-26 16:25:45,721 DEBUG:Retry: /2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch
2024-02-26 16:25:45,926 DEBUG:https://api.bitbucket.org:443 "GET /2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch HTTP/1.1" 429 46
2024-02-26 16:25:45,926 DEBUG:Incremented Retry for (url='/2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch'): Retry(total=None, connect=None, read=None, redirect=None, status=998)
2024-02-26 16:25:48,230 DEBUG:Retry: /2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch
2024-02-26 16:25:48,473 DEBUG:https://api.bitbucket.org:443 "GET /2.0/repositories/<workspace>/<repo>/refs/branches/flichtenheld/branch HTTP/1.1" 200 None
2024-02-26 16:25:48,474 DEBUG:HTTP: GET flichtenheld/branch -> 200 OK
...
codecov-commenter commented 4 months ago

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 34.16%. Comparing base (5c54e5d) to head (b2383e6). Report is 1 commits behind head on master.

Files Patch % Lines
atlassian/rest_client.py 40.00% 2 Missing and 1 partial :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #1339 +/- ## ======================================= Coverage 34.16% 34.16% ======================================= Files 45 45 Lines 8368 8373 +5 Branches 1176 1177 +1 ======================================= + Hits 2859 2861 +2 - Misses 5395 5397 +2 - Partials 114 115 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

gonchik commented 4 months ago

@flichtenheld Thanks!