Closed jsmatias closed 6 months ago
Currently there's no check on the rate limits during synchronisation process. As it doesn't stop upon error 429, it skips many assets.
2024-01-29 13:39:19.567 ERROR synchronization - main: Error on identifier zenodo.org:8263868 Traceback (most recent call last): File "/app/connectors/zenodo/zenodo_datasetconnector.py", line 216, in fetch yield datetime, self._dataset_fromrecord(id, resource) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/connectors/zenodo/zenodo_dataset_connector.py", line 154, in _dataset_from_record response.raise_for_status() File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: TOO MANY REQUESTS for url: https://zenodo.org/api/records/8263868/files
The rate limits are available on zenodo API documentation: https://developers.zenodo.org/#rate-limiting
Currently there's no check on the rate limits during synchronisation process. As it doesn't stop upon error 429, it skips many assets.
2024-01-29 13:39:19.567 ERROR synchronization - main: Error on identifier zenodo.org:8263868 Traceback (most recent call last): File "/app/connectors/zenodo/zenodo_datasetconnector.py", line 216, in fetch yield datetime, self._dataset_fromrecord(id, resource) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/connectors/zenodo/zenodo_dataset_connector.py", line 154, in _dataset_from_record response.raise_for_status() File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: TOO MANY REQUESTS for url: https://zenodo.org/api/records/8263868/files
The rate limits are available on zenodo API documentation: https://developers.zenodo.org/#rate-limiting