search

codeforIATI / iatikit

🐨 A toolkit for using IATI data
https://iatikit.readthedocs.io
MIT License
6 stars 0 forks source link

Add retries to download functions #58

Closed tillywoodfield closed 6 months ago

tillywoodfield commented 6 months ago

Is your feature request related to a problem? Please describe.

When calling functions from utils.download, occasionally we see a connection error which resolves itself on a manual rerun.

Describe the solution you’d like

Add a retry around all of the top-level functions from utils.download.

Describe alternatives you’ve considered

We can add a retry in the application which is using iatikit, but think it would be more useful to add it to iatikit directly so that others can benefit from it.

Additional context

Example of the connection error:

Traceback (most recent call last):
  File "/home/iatitables/iatitables/runner.py", line 6, in <module>
    iatidata.run_all(processes=1)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 1368, in run_all
    process_registry(refresh=refresh, sample=sample, processes=processes)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 335, in process_registry
    save_all(sample=sample, parts=processes, refresh=refresh)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 308, in save_all
    get_standard(refresh)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 101, in get_standard
    iatikit.download.standard()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 250, in standard
    codelists()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 187, in codelists
    version_codelist = get_codelist(codelist_name, version)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 165, in get_codelist
    version_codelist = requests.get(codelist_url).json()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in send
    history = [resp for resp in gen]
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in <listcomp>
    history = [resp for resp in gen]
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
andylolz commented 6 months ago

Fixed in #59.