The connector was unable to fetch all records and process them before the resumption token expiration. (422 error from zenodo). This is related to the issue #267
Solution: I separated fetching all records from processing them, and added a timer to stop fetching before the given expiration.
The previous bug resulted in skipping several records.
Solution: I pass the timestamp of the last processed record to the state. This way the next iteration will start from the last record, instead of to_excl_current, which is defined as next cycle (previous from_incl + time_per_loop)
Zenodo connector was exceeding the request rate limits of the Zenodo platform. This is related to the issue #267
Solution: I used decorators on the fetching methods to define the limits and sleep and retry to avoid raising the 429 error.
Many KeyErrors due to missing fields were not properly addressed. This were all passed to the log files making it less readable to debug and quickly overloads the log files.
Solution: I used the dictionary .get() method to return a None or default value to be evaluated subsequently.
Changes:
I included a few unit tests to test these bugfixes
Changed the initial date from which the connector starts fetching data 2023 -> 2020
Bugfixes:
to_excl_current, which is defined as next cycle (previousfrom_incl + time_per_loop)429 error.KeyErrors due to missing fields were not properly addressed. This were all passed to the log files making it less readable to debug and quickly overloads the log files. Solution: I used the dictionary.get()method to return aNoneor default value to be evaluated subsequently.Changes: