Investigate partition "changes" when updating FERC archives

catalyst-cooperative / pudl-archiver

A tool for capuring snapshots of public data sources and archiving them on Zenodo for programmatic use.

MIT License

4 stars 1 forks source link

Investigate partition "changes" when updating FERC archives #399

Closed e-belfer closed 1 week ago

e-belfer commented 3 months ago

In #398 and the corresponding archive run, every FERC XBRL archive was marked as having its partition changed, even though the links are identical to what they were. In September, if this happens again, we should investigate it. This is a two-fold task:

Investigate the "changed" partitions
Sort the values in the taxonomies_zip, to prevent them from changing each month by order.

zschira commented 1 month ago

I've opened a PR to attempt to address this issue. See #433 for more details.

zschira commented 1 week ago

Closing. Partition changes are fixed, but there's still issues with XBRL zip file changes which will be tracked in #471.