Is your feature request related to a problem? Please describe.
When calling functions from utils.download, occasionally we see a connection error which resolves itself on a manual rerun.
Describe the solution you’d like
Add a retry around all of the top-level functions from utils.download.
Describe alternatives you’ve considered
We can add a retry in the application which is using iatikit, but think it would be more useful to add it to iatikit directly so that others can benefit from it.
Additional context
Example of the connection error:
Traceback (most recent call last):
File "/home/iatitables/iatitables/runner.py", line 6, in <module>
iatidata.run_all(processes=1)
File "/home/iatitables/iatitables/iatidata/__init__.py", line 1368, in run_all
process_registry(refresh=refresh, sample=sample, processes=processes)
File "/home/iatitables/iatitables/iatidata/__init__.py", line 335, in process_registry
save_all(sample=sample, parts=processes, refresh=refresh)
File "/home/iatitables/iatitables/iatidata/__init__.py", line 308, in save_all
get_standard(refresh)
File "/home/iatitables/iatitables/iatidata/__init__.py", line 101, in get_standard
iatikit.download.standard()
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 250, in standard
codelists()
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 187, in codelists
version_codelist = get_codelist(codelist_name, version)
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 165, in get_codelist
version_codelist = requests.get(codelist_url).json()
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in send
history = [resp for resp in gen]
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in <listcomp>
history = [resp for resp in gen]
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 266, in resolve_redirects
resp = self.send(
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Is your feature request related to a problem? Please describe.
When calling functions from
utils.download
, occasionally we see a connection error which resolves itself on a manual rerun.Describe the solution you’d like
Add a retry around all of the top-level functions from
utils.download
.Describe alternatives you’ve considered
We can add a retry in the application which is using iatikit, but think it would be more useful to add it to iatikit directly so that others can benefit from it.
Additional context
Example of the connection error: