Fix AEMET API assets - Githubissues

datonic / datadex

📦 Serverless and local-first Open Data Platform

http://datadex.datonic.io

MIT License

220 stars 14 forks source link

Fix AEMET API assets #60

Closed davidgasquez closed 1 month ago

davidgasquez commented 1 month ago

Seems the GitHub CI machine the project has been assigned recently keeps crashing due to AEMET rate limiting the machine itself.

This shouldn't happens, and if it happens, we should handle it better than crashing the entire DAG.

davidgasquez commented 1 month ago

Thinking about creating a PandasHuggingFaceIOManager and figure out it an asset can load itself from there and continue with incremental requests to the API instead of the entire date range.

davidgasquez commented 1 month ago

Issue might be the stability of the servers. I've been getting this even doing manual queries... :sweat_smile: