Processing six-month historical datasets from the EIA GridMonitor

[jdechalendar]

Commit summary

commit a3cc178d943de610aed76ea99596ade469fa9288
Date:   Sun Dec 31 19:10:15 2023 +0100

    Process six-month files from EIA Grid Monitor

    * Download script

    * Parsers for six month bulk files - Implemented in eia_bulk_grid_monitor.py

    * Script to parse/process EIA Grid Monitor in bulk

    * bugfix(bulk_process): concat should use axis=1

    * Moved EIA_Grid_Monitor downloads to a downloads folder

    * Updates to Makefile and requirements in setup.py

    * Added notebook to demo usage of bulk data downloads

    * Update to viz.reports to use GraphData

    * update README

commit 3e26cbc21ab2d1d99e18a8b88c1bbb94a2079fd8
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Mon Jan 1 11:41:48 2024 +0100

    workflows.make_dataset minor updates

    * Signature change: start and end are now optional

    * Docstring update

Sample output from sanity check notebook

[jdechalendar]

@ktehranchi @JuliaF556 I think this is almost ready to be closed. Bulk dataset is available here. I also added this notebook on the branch for this PR with my quick sanity check. See plots in PR description.

Could you both try using these updated datasets for your respective projects. If the data seem to be what you expect I will close this PR. I am keeping track of unfinished work I still want to do for this in #22

[JuliaF556]

The bulk download works for me, so I'm just struggling with the new EIA format and will build a work-around to construct a full timeseries based on the old format for my project.

[ktehranchi]

Data looks good on my end. Will you be keeping this format of the data? ie separated into the equivalent 6 mo files?

[jdechalendar]

Data looks good on my end. Will you be keeping this format of the data? ie separated into the equivalent 6 mo files?

Planning on also merging into one file with everything -- but haven't gotten around to doing that yet.

[JuliaF556]

I built a workaround to append the new 6-month data to my stored dataset (2015-February 2023). It might be helpful, but I'm keeping everything in the old naming format. I'm also dropping columns that weren't reported in EIA930 before the naming change.

[ktehranchi]

Hey Jacques - I just noticed the minimum of all the energy carriers was ~1.0.... Is there a reason why? Should I subtract 1 from all the data?... I guess its such a small number it shouldn't make a difference.

[grgmiller]

See: https://github.com/singularity-energy/open-grid-emissions/issues/113 and https://github.com/jdechalendar/gridemissions/issues/8

I think this is a placeholder for missing data