search

spacetelescope / jirahub

A package for syncing JIRA tickets and Github issues
http://jirahub.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
14 stars 8 forks source link

Handle Server Timeouts and 503 responses #21

Open jbcurtin opened 5 years ago

jbcurtin commented 5 years ago

Stacktrace

Traceback (most recent call last):
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/adapters.py", line 440, in send
    timeout=timeout
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/spacetelescope/jwst/issues?since=2019-09-21T13%3A02%3A32Z (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0c9d4887f0>, 'Connection to api.github.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "jwst_pipeline_jgbot.py", line 162, in <module>
    issues = clean_issues(g, j, issues, args.clean)
  File "jwst_pipeline_jgbot.py", line 98, in clean_issues
    gh_issue_list = [str(github_issue.number) for github_issue in github_updated_issues]
  File "jwst_pipeline_jgbot.py", line 98, in <listcomp>
    gh_issue_list = [str(github_issue.number) for github_issue in github_updated_issues]
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/PaginatedList.py", line 58, in __iter__
    newElements = self._grow()
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/PaginatedList.py", line 70, in _grow
    newElements = self._fetchNextPage()
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/PaginatedList.py", line 172, in _fetchNextPage
    headers=self.__headers
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/Requester.py", line 258, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers, input, cnx))
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/Requester.py", line 304, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/Requester.py", line 357, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/Requester.py", line 384, in __requestRaw
    response = cnx.getresponse()
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/PyGithub-1.40a3-py3.6.egg/github/Requester.py", line 105, in getresponse
    r = verb(url, headers=self.headers, data=self.input, timeout=self.timeout, verify=self.verify)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/adapters.py", line 496, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/spacetelescope/jwst/issues?since=2019-09-21T13%3A02%3A32Z (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0c9d4887f0>, 'Connection to api.github.com timed out. (connect timeout=10)'))
Sat Sep 21 14:37:02 EDT 2019
Sat Sep 21 15:00:03 EDT 2019
Sat Sep 21 15:37:02 EDT 2019
Sat Sep 21 16:00:03 EDT 2019
Sat Sep 21 16:37:03 EDT 2019
Sat Sep 21 17:00:03 EDT 2019
Sat Sep 21 17:37:02 EDT 2019
Sat Sep 21 18:00:03 EDT 2019
Sat Sep 21 18:37:02 EDT 2019
WARNING:root:Got recoverable error from GET https://jira.stsci.edu/rest/api/2/serverInfo, will retry [1/3] in 3.5776933550275625s. Err: 503 Service Temporarily Unavailable
WARNING:root:Got recoverable error from GET https://jira.stsci.edu/rest/api/2/serverInfo, will retry [2/3] in 25.177439455726436s. Err: 503 Service Temporarily Unavailable
WARNING:root:Got recoverable error from GET https://jira.stsci.edu/rest/api/2/serverInfo, will retry [3/3] in 4.9389332646908635s. Err: 503 Service Temporarily Unavailable
Traceback (most recent call last):
  File "jda_jgbot.py", line 175, in <module>
    create_issues(issues, jirarepo, jirauser, jirapass, gitkey)
  File "jda_jgbot.py", line 104, in create_issues
    j = JiraQuery(jirarepo, user=jirauser, password=jirapass)
  File "./programs/jirahub/jirahub/jiraquery.py", line 39, in __init__
    self.jira = JIRA(site, basic_auth=(user, password))
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/client.py", line 465, in __init__
    si = self.server_info()
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/client.py", line 2100, in server_info
    j = self._get_json('serverInfo')
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/client.py", line 2516, in _get_json
    r = self._session.get(url, params=params)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/resilientsession.py", line 151, in get
    return self.__verb('GET', url, **kwargs)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/resilientsession.py", line 147, in __verb
    raise_on_error(response, verb=verb, **kwargs)
  File "./anaconda3/envs/jirahub/lib/python3.6/site-packages/jira-1.0.15-py3.6.egg/jira/resilientsession.py", line 57, in raise_on_error
    r.status_code, error, r.url, request=request, response=r, **kwargs)
jira.exceptions.JIRAError: JiraError HTTP 503 url: https://jira.stsci.edu/rest/api/2/serverInfo
    text: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

    response headers = {'Date': 'Sat, 21 Sep 2019 22:40:06 GMT', 'Content-Length': '323', 'Connection': 'close', 'Content-Type': 'text/html; charset=iso-8859-1'}
    response text = <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>
stscijgbot commented 5 years ago

Tracked on JIRA as issue SCSB-22.

eslavich commented 5 years ago

The underlying JIRA client has backoff-and-retry built in, which we can configure with jirahub's c.jira.max_retries parameter. Do you think it's reasonable to "solve" this by simply increasing the number of retries?

This is their backoff implementation, btw, it looks solid: https://github.com/pycontribs/jira/blob/ec0432b852448d2fcd61f83fa581af6575f0c67c/jira/resilientsession.py#L100