The OpenML connector reaches last asset while syncing and tries to get the next one incrementing the previous identifier.
As it doesn't exist, OpenML raises an error on null identifier and the connector skips this asset.
The issue is that it keeps incrementing the offset every time that it reaches the end of the list.
If no asset is added to OpenML, the offset will keep increasing indefinitely, while it should be the number of existing assets on OpenML during the the last synchronisation.
Ex.: Assets on OpenML: 3
When the connector tries to fetch the metadata from the 4th asset, if gets an error and stops, but the offset is set to 4, instead of 3. During next run, it will try start synchronising from asset 4, as it doesn't exist, it stops and increases the offset again in 1 unit (to 5).
Error from the synchronisation logs:
...
2024-02-06 16:25:03.005 ERROR synchronization - main: Error on identifier None: Error while fetching https://www.openml.org/api/v1/json/data/list/limit/500/offset/**5435 from OpenML: 'No results'.
2024-02-06 16:25:03.008 INFO resource_connector_by_id - run: Finished: 1 < 500
2024-02-06 20:25:02.639 INFO synchronization - main: Done
...
2024-02-06 20:25:02.633 ERROR synchronization - main: Error on identifier None: Error while fetching https://www.openml.org/api/v1/json/data/list/limit/500/offset/5436** from OpenML: 'No results'.
2024-02-06 20:25:02.637 INFO resource_connector_by_id - run: Finished: 1 < 500
2024-02-06 20:25:02.639 INFO synchronization - main: Done
Solution: This can be fully addressed with what is suggested at issue #247. However, a quick solution is to recognise the end of the list and stop increasing the offset.
The OpenML connector reaches last asset while syncing and tries to get the next one incrementing the previous identifier. As it doesn't exist, OpenML raises an error on
nullidentifier and the connector skips this asset. The issue is that it keeps incrementing the offset every time that it reaches the end of the list. If no asset is added to OpenML, the offset will keep increasing indefinitely, while it should be the number of existing assets on OpenML during the the last synchronisation.Ex.: Assets on OpenML: 3
When the connector tries to fetch the metadata from the 4th asset, if gets an error and stops, but the offset is set to 4, instead of 3. During next run, it will try start synchronising from asset 4, as it doesn't exist, it stops and increases the offset again in 1 unit (to 5).
Error from the synchronisation logs: ... 2024-02-06 16:25:03.005 ERROR synchronization - main: Error on identifier None: Error while fetching https://www.openml.org/api/v1/json/data/list/limit/500/offset/**5435 from OpenML: 'No results'. 2024-02-06 16:25:03.008 INFO resource_connector_by_id - run: Finished: 1 < 500 2024-02-06 20:25:02.639 INFO synchronization - main: Done ... 2024-02-06 20:25:02.633 ERROR synchronization - main: Error on identifier None: Error while fetching https://www.openml.org/api/v1/json/data/list/limit/500/offset/5436** from OpenML: 'No results'. 2024-02-06 20:25:02.637 INFO resource_connector_by_id - run: Finished: 1 < 500 2024-02-06 20:25:02.639 INFO synchronization - main: Done
Solution: This can be fully addressed with what is suggested at issue #247. However, a quick solution is to recognise the end of the list and stop increasing the offset.