search

biglocalnews / bln-python-client

Python client for the biglocalnews.org API
https://bln-python-client.readthedocs.io
Apache License 2.0
5 stars 5 forks source link

Consider retry on failed downloads #85

Closed stucka closed 1 year ago

stucka commented 1 year ago

Run pipenv run warn-transformer download -l DEBUG 2023-08-05 00:19:44,440 - warn_transformer.cli - Running download command Traceback (most recent call last): File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/bin/warn-transformer", line 8, in sys.exit(cli()) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 783, in invoke return __callback(args, **kwargs) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn_transformer/cli.py", line 48, in download download_runner.run(download_dir, source) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn_transformer/download.py", line 32, in run p = c.get_project_by_name("WARN Act Notices") File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 312, in get_project_by_name project_list = self.search_projects(lambda x: x["name"] == name) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 404, in search_projects for v in self.effectiveProjectRoles(): File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 95, in effectiveProjectRoles return self._gql(q.query_effectiveProjectRoles) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 54, in _gql raise APIException(err) bln.exceptions.APIException: Internal Server Error Error: Process completed with exit code 1.

stucka commented 1 year ago

Error happening in production, also:

2023-08-08 00:20:40,169 - warn_transformer.download - Download amendments.csv to data/warn-transformer/raw 2023-08-08 00:20:41,883 - warn_transformer.download - Download ny.csv to data/warn-transformer/raw 2023-08-08 00:20:43,604 - warn_transformer.download - Download tn.csv to data/warn-transformer/raw 2023-08-08 00:20:45,335 - warn_transformer.download - Download ia.csv to data/warn-transformer/raw ... File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn_transformer/cli.py", line 48, in download download_runner.run(download_dir, source) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn_transformer/download.py", line 48, in run c.download_file(BLN_PROJECT_ID, f, output_dir=download_dir) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 339, in download_file uri = self.createFileDownloadUri(projectId, filename) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 115, in createFileDownloadUri return self._gql(q.mutation_createFileDownloadUri, locals()) File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/bln/client.py", line 54, in _gql raise APIException(err) bln.exceptions.APIException: Internal Server Error Error: Process completed with exit code 1.

stucka commented 1 year ago

First failure was a success.

2023-08-09 10:05:02,280 - warn_transformer.download - Download nj.csv to /home/stucka/.warn-transformer/raw 2023-08-09 10:05:04,267 - warn_transformer.download - Download sc.csv to /home/stucka/.warn-transformer/raw 2023-08-09 10:05:08,615 - retry.api - Internal Server Error, retrying in 10 seconds... 2023-08-09 10:05:21,540 - warn_transformer.download - Download de.csv to /home/stucka/.warn-transformer/raw

stucka commented 1 year ago

Should be closed with https://github.com/biglocalnews/bln-python-client/compare/2.3.2...2.3.3