search

nodiscc / hecat

Generic automation tool around data stored as plaintext YAML files
GNU General Public License v3.0
29 stars 5 forks source link

processors/github_metadata: retry in case of API request timeout #114

Open nodiscc opened 1 year ago

nodiscc commented 1 year ago

Would prevent having to manually re-run the whole job, e.g. https://github.com/awesome-selfhosted/awesome-selfhosted-data/actions/runs/5662427414/job/15342323076

[...]
INFO:github_metadata.py: Gathering metadata for https://github.com/revive-adserver/revive-adserver from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/revoltchat/revolt from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/roadiz/roadiz from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/robustirc/robustirc from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/RocketChat/Rocket.Chat from Github API
Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/bin/hecat", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/main.py", line 39, in main
    add_github_metadata(step)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/processors/github_metadata.py", line 107, in add_github_metadata
    gh_metadata, latest_commit_date = get_gh_metadata(step, github_url, g, errors)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/processors/github_metadata.py", line 59, in get_gh_metadata
    gh_metadata = g.get_repo(project)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/MainClass.py", line 363, in get_repo
    headers, data = self.__requester.requestJsonAndCheck("GET", url)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 443, in requestJsonAndCheck
    *self.requestJson(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 580, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 685, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 719, in __requestRaw
    response = cnx.getresponse()
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 164, in getresponse
    r = verb(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)
make: *** [Makefile:25: update_metadata] Error 1
Error: Process completed with exit code 2.