Create a github action to update and sync the datastore caches

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

456 stars 106 forks source link

Create a github action to update and sync the datastore caches #1895

Closed bendnorman closed 1 year ago

bendnorman commented 1 year ago

Create a github action that runs the datastore script to update the GCS cache if any new DOIs have been added to the code. The action should also sync the internal-zenodo-cache and zenodo-cache buckets.

@zaneselvans should this action checkout dev or main for the updated DOIs in the datastore script?

https://github.com/catalyst-cooperative/pudl/blob/d294cb52b62b6adf18384b2bbb1be15c84667e6f/src/pudl/workspace/datastore.py#L140-L159

I think dev so nightly builds won't have to populate the GCS cache.

TODOs

[x] Create a service account for the action with permission to read and write to the internal and public cache
[x] Create steps to setup pudl environment
[x] Step to run the pudl_datastore command to update the internal cache
[x] Step to authenticate gcloud command
[x] rsync internal cache to public cache

zaneselvans commented 1 year ago

I think it has to check out dev otherwise it'll never update anything that's needed for the nightly builds, which run on dev. Right?

bendnorman commented 1 year ago

Groovy, just wanted to double-check.

zaneselvans commented 1 year ago

It's a whole separate versioning regime really. At some point I think we should have 2 actions running nightly:

First: one that scrapes the publishing agency websites, and creates new candidate archives to upload, and if there are any new / changed files (based on their hashes) it creates new draft Zenodo depositions and notifies us to review and click "publish"
Second: one that has a list of "concept DOIs" it checks every night for new versions of the data (each concept DOI identifies an archive lineage, e.g. ferc1 or eia923). If there are new versions in any of the archive lineages it tracks, then it uploads them into the internal zenodo cache, and rsyncs that to the public bucket.

Then we'll have:

Every version of the data ever published archived.
Fresh copies of that data available for the ETL to use in GCS, regardless of what particular DOI it's referring to.

bendnorman commented 1 year ago

Love this idea. Once we get more familiar with Dagster Cloud we could also take advantage of sensors to run both of these workflows.