Cache eia api v2 - Githubissues

TrentonBush commented 2 years ago

Draft implementation of EIA API downloader. There are a few warts and works in progress:

no http error handling - currently a single bad request will grind everything to a halt.
It is sloooow. Downloading the monthly series took me a little over an hour. This is due to the glacial response time of the EIA API. Each request takes several seconds and is capped at 5k records. There are nearly 4 million records to download (mostly nulls). We could switch to async but EIA threatens to suspend/ban API keys for exceeding rate limits (and doesn't say anywhere what the limits are 🙄).
the code is ugly! I think there could be much cleaner separation of disk IO and networking.

I'm wondering if it would be easier to use the bulk download and cut unneeded things than to fix this up. I think it would be more reliable, faster to download, faster to develop, and simpler, at the cost of some bandwidth and (temporary!) disk space.

zaneselvans commented 2 years ago

Welp, okay. An hour to download one small subset of the electricity data seems absurd. We should verify that the bulk download has the same information (hopefully they update it automatically...) and just use that. I was able to download the entire 192MB zipped JSON archive in 26 seconds in Mexico over a VPN so.... yeah. I guess we'll just archive that instead and subsequently extract the series we need. 🙄 But at least we'll have all of them! Assuming it's automatically updated and really contains whatever is inside the v2 API. Have you looked at its contents already?

zaneselvans commented 2 years ago

I've got the full eia_bulk_elec data source in the datastore now, so I think this can be closed.

catalyst-cooperative / pudl-scrapers

Cache eia api v2 #22