search

pycontribs / jira

Python Jira library. Development chat available on https://matrix.to/#/#pycontribs:matrix.org
https://jira.readthedocs.io
BSD 2-Clause "Simplified" License
1.94k stars 860 forks source link

Handler for HTTP 429 Too Many Requests #1657

Open D-Mielewczyk opened 1 year ago

D-Mielewczyk commented 1 year ago

Problem trying to solve

My integration tests regarding a JIRA bot, are sending too many requests and I am getting this exception:

[2023-05-16T13:22:56.021Z] tests/test_integration_stella.py:34: 
[2023-05-16T13:22:56.021Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2023-05-16T13:22:56.021Z] .tox/integrate/lib/python3.6/site-packages/jira/resources.py:741: in delete
[2023-05-16T13:22:56.021Z]     super().delete(params={"deleteSubtasks": deleteSubtasks})
[2023-05-16T13:22:56.022Z] .tox/integrate/lib/python3.6/site-packages/jira/resources.py:443: in delete
[2023-05-16T13:22:56.022Z]     return self._session.delete(url=self.self, params=params)
[2023-05-16T13:22:56.022Z] .tox/integrate/lib/python3.6/site-packages/jira/resilientsession.py:204: in delete
[2023-05-16T13:22:56.022Z]     return self.__verb("DELETE", str(url), **kwargs)
[2023-05-16T13:22:56.022Z] .tox/integrate/lib/python3.6/site-packages/jira/resilientsession.py:189: in __verb
[2023-05-16T13:22:56.022Z]     raise_on_error(response, verb=verb, **kwargs)
[2023-05-16T13:22:56.022Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2023-05-16T13:22:56.022Z] 
[2023-05-16T13:22:56.022Z] r = <Response [429]>, verb = 'DELETE'
[2023-05-16T13:22:56.022Z] kwargs = {'headers': {'User-Agent': 'python-requests/2.27.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json,*... 'no-cache', 'Content-Type': 'application/json', 'X-Atlassian-Token': 'no-check'}, 'params': {'deleteSubtasks': False}}
[2023-05-16T13:22:56.022Z] request = None, error = 'Rate limit exceeded.'
[2023-05-16T13:22:56.022Z] response = {'message': 'Rate limit exceeded.'}
[2023-05-16T13:22:56.022Z] 
[2023-05-16T13:22:56.022Z]     def raise_on_error(r: Optional[Response], verb="???", **kwargs):
[2023-05-16T13:22:56.022Z]         """Handle errors from a Jira Request
[2023-05-16T13:22:56.022Z]     
[2023-05-16T13:22:56.022Z]         Args:
[2023-05-16T13:22:56.022Z]             r (Optional[Response]): Response from Jira request
[2023-05-16T13:22:56.022Z]             verb (Optional[str]): Request type, e.g. POST. Defaults to "???".
[2023-05-16T13:22:56.022Z]     
[2023-05-16T13:22:56.022Z]         Raises:
[2023-05-16T13:22:56.022Z]             JIRAError: If Response is None
[2023-05-16T13:22:56.022Z]             JIRAError: for unhandled 400 status codes.
[2023-05-16T13:22:56.022Z]             JIRAError: for unhandled 200 status codes.
[2023-05-16T13:22:56.022Z]         """
[2023-05-16T13:22:56.022Z]         request = kwargs.get("request", None)
[2023-05-16T13:22:56.022Z]         # headers = kwargs.get('headers', None)
[2023-05-16T13:22:56.022Z]     
[2023-05-16T13:22:56.022Z]         if r is None:
[2023-05-16T13:22:56.022Z]             raise JIRAError(None, **kwargs)
[2023-05-16T13:22:56.022Z]     
[2023-05-16T13:22:56.022Z]         if r.status_code >= 400:
[2023-05-16T13:22:56.022Z]             error = ""
[2023-05-16T13:22:56.022Z]             if r.status_code == 403 and "x-authentication-denied-reason" in r.headers:
[2023-05-16T13:22:56.022Z]                 error = r.headers["x-authentication-denied-reason"]
[2023-05-16T13:22:56.022Z]             elif r.text:
[2023-05-16T13:22:56.022Z]                 try:
[2023-05-16T13:22:56.022Z]                     response = json.loads(r.text)
[2023-05-16T13:22:56.022Z]                     if "message" in response:
[2023-05-16T13:22:56.022Z]                         # Jira 5.1 errors
[2023-05-16T13:22:56.022Z]                         error = response["message"]
[2023-05-16T13:22:56.022Z]                     elif "errorMessages" in response and len(response["errorMessages"]) > 0:
[2023-05-16T13:22:56.022Z]                         # Jira 5.0.x error messages sometimes come wrapped in this array
[2023-05-16T13:22:56.022Z]                         # Sometimes this is present but empty
[2023-05-16T13:22:56.022Z]                         errorMessages = response["errorMessages"]
[2023-05-16T13:22:56.022Z]                         if isinstance(errorMessages, (list, tuple)):
[2023-05-16T13:22:56.022Z]                             error = errorMessages[0]
[2023-05-16T13:22:56.022Z]                         else:
[2023-05-16T13:22:56.022Z]                             error = errorMessages
[2023-05-16T13:22:56.022Z]                     # Catching only 'errors' that are dict. See https://github.com/pycontribs/jira/issues/350
[2023-05-16T13:22:56.022Z]                     elif (
[2023-05-16T13:22:56.022Z]                         "errors" in response
[2023-05-16T13:22:56.022Z]                         and len(response["errors"]) > 0
[2023-05-16T13:22:56.022Z]                         and isinstance(response["errors"], dict)
[2023-05-16T13:22:56.022Z]                     ):
[2023-05-16T13:22:56.022Z]                         # Jira 6.x error messages are found in this array.
[2023-05-16T13:22:56.022Z]                         error_list = response["errors"].values()
[2023-05-16T13:22:56.022Z]                         error = ", ".join(error_list)
[2023-05-16T13:22:56.022Z]                     else:
[2023-05-16T13:22:56.022Z]                         error = r.text
[2023-05-16T13:22:56.022Z]                 except ValueError:
[2023-05-16T13:22:56.022Z]                     error = r.text
[2023-05-16T13:22:56.022Z]             raise JIRAError(
[2023-05-16T13:22:56.022Z]                 error,
[2023-05-16T13:22:56.022Z]                 status_code=r.status_code,
[2023-05-16T13:22:56.022Z]                 url=r.url,
[2023-05-16T13:22:56.022Z]                 request=request,
[2023-05-16T13:22:56.022Z]                 response=r,
[2023-05-16T13:22:56.022Z] >               **kwargs,
[2023-05-16T13:22:56.022Z]             )
[2023-05-16T13:22:56.022Z] E           jira.exceptions.JIRAError: JiraError HTTP 429 url: https://jira.com/rest/api/2/issue/16664769?deleteSubtasks=False
[2023-05-16T13:22:56.022Z] E            text: Rate limit exceeded.
[2023-05-16T13:22:56.022Z] E            
[2023-05-16T13:22:56.023Z] E            response headers = {'X-AREQUESTID': '982x5431467x7', 'X-ANODEID': 'node3', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': 'sandbox', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'JSESSIONID=node3~CA507D81DCCBDECECC9A42AC867864AD; Path=/; Secure; HttpOnly, atlassian.xsrf.token=BACO-OOY2-7ZTP-186A_3fd5f15ab3dbe9f7aacddd6e5b248651e9563f7e_lin; Path=/; Secure; SameSite=None', 'X-Seraph-LoginReason': 'OK', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '0', 'X-RateLimit-FillRate': '60', 'X-RateLimit-Interval-Seconds': '60', 'Retry-After': '0', 'Content-Encoding': 'gzip', 'Vary': 'User-Agent', 'Content-Type': 'application/json;charset=UTF-8', 'Content-Length': '54', 'Date': 'Tue, 16 May 2023 13:22:52 GMT'}
[2023-05-16T13:22:56.023Z] E            response text = {"message":"Rate limit exceeded."}
[2023-05-16T13:22:56.023Z] 
[2023-05-16T13:22:56.023Z] .tox/integrate/lib/python3.6/site-packages/jira/resilientsession.py:70: JIRAError

Possible solution(s)

I wish that this library could retry requests that were met with HTTP 429 response after a certain amount of time.

Alternatives

Currently, I am using retry module alongside with some ugly time.sleep() lines. If anyone has a better workaround I would be happy to see it.

Additional Context

No response

studioj commented 1 year ago

@D-Mielewczyk thank you for your issue submission. simple first suggestion use the delay part in the retry module. secondly we're always open for suggestions via a Pull Request. Implementing a fallback in case of a 429 could be useful. Don't forget to add a test (probably mocking the 429 response would be needed) kind regards