search

catalyst-cooperative / pudl-archiver

A tool for capuring snapshots of public data sources and archiving them on Zenodo for programmatic use.
MIT License
4 stars 2 forks source link

Archive FERC EQR #31

Closed katie-lamb closed 11 months ago

katie-lamb commented 1 year ago

There's high demand for FERC EQR data. Even if we haven't integrated it yet, we can still archive it.

FERC EQR data.

This is a duplicate issue of PUDL issue 1414, but I'm putting it in the archiver repo.

e-belfer commented 1 year ago

I can have a go at it.

zaneselvans commented 1 year ago

The EQR may be a bit weird because it's big and also comes in multiple formats. GitHub runners only have 14GB of disk but I think the whole EQR dataset is significantly larger than that (though each individual file won't be).

They don't make the download pages easy to find. The new format (2013-present) uses a kind of annoying javascripty interface but seems to have direct links embedded in it that are formulaic. The older data (pre-2013) splits out transaction tables into a bunch of partitions, and then has the non-trasaction tables for all years in its own separate file.

Right now it looks like each quarterly zipped CSV is around 3GB. So even 1 year might exceed the a GitHub runner's disk space. I wonder how hard / worthwhile it would be to make it possible to serialize the download / upload process from FERC to the runner and then up to Zenodo so the whole dataset isn't ever on disk at the same time.

At least they've finally shut down the janky FTP site and now these can all be downloaded over HTTPS!

e-belfer commented 1 year ago

Thanks for the heads up, Zane. Just talked to Zach & Katie about how to structure this. Github runner questions outstanding, I'll start on the archiver part of this, following the FERC2 archiver for some inspiration on how to handle year-to-year file structure changes.

zaneselvans commented 1 year ago

IIRC Zenodo also has a 50GB per deposition limit, but like AWS Open Data it's possible to request a higher quota.

jdangerx commented 1 year ago

As a shorter workaround than #44, it should be pretty quick to add a command line flag to restrict EQR to certain partitions - and then we could just split the EQR archival process across like 4 different runners in the actions config.

bendnorman commented 10 months ago

I see the archive is on sandbox.zenodo.org. Should it also be available on zenodo.org?