search

singularity-energy / open-grid-emissions

Tools for producing high-quality hourly generation and emissions data for U.S. electric grids
MIT License
72 stars 5 forks source link

Add option to use pudl nightly builds #284

Closed grgmiller closed 9 months ago

grgmiller commented 1 year ago

When running the OGE pipeline, we currently use the most recent versioned release of PUDL available on Zenodo. However, pudl is now publishing nightly builds of their data here: https://github.com/catalyst-cooperative/pudl#nightly-data-builds. This could allow us to use fresher data for testing purposes, and if we ever wanted to be able to use monthly data files (like https://github.com/singularity-energy/open-grid-emissions/issues/212).

To implement, we might just want to add a command line option to allow the user to specify where to download the pudl data from.

grgmiller commented 1 year ago

So after talking to the folks at Catalyst, it sounds like in addition to running nightly builds, they will also be doing more frequent/automatic versioned releases in zenodo. However, these zenodo releases will also be paired to a persistent version of the data on amazon cloud storage.

So in terms of next steps, we may want to:

  1. Change the dependency to grab the data from the cloud storage instead of zenodo, and figure out a way to automatically notify us when a new version is released.
  2. Set up a development version of OGE that uses the nightly builds and monitors that for changes.
grgmiller commented 9 months ago

Addressed by https://github.com/singularity-energy/open-grid-emissions/pull/318