search

jdechalendar / gridemissions

Tools for power sector emissions tracking
MIT License
37 stars 6 forks source link

Various updates to plotting utils and dataset convenience tools #42

Closed jdechalendar closed 5 months ago

jdechalendar commented 5 months ago

Summary of changes

Changes made by our data processing tools to the raw datasets

Below is some output from the corresponding jupyter notebooks. Future work: add some of this content to energy.stanford.edu/gridemissions.

Note: before running these notebooks, you need to download the bulk datasets, which you can now do with ge_download --bulk.

Figs 2-5 from de Chalendar and Benson 2021.ipynb

Paper on the arXiv. I added dropdowns to slice data by year.

Figure 2 from the paper for 2018 below. download-1

Figure 2 from the paper for 2023 below. download

Figures 3-5 (not shown here but available in the notebook) summarize adjustments for interchange data and generation by source.

Cleaning plot.ipynb

gridemissions.viz.reports.cleaning_plot is a plotting function I used when I initially developed this tool. I updated it as part of this PR and am showing how I use it in a notebook. BPAT is one of the balancing areas whose data is most changed by the data reconciliation algorithm (I have not thoroughly investigated why).

The cleaning_plot function has different options. I am showing two options below for the BPAT example. The red line in the first subplot below indicates a large mismatch (up to 5GW) between supply, demand and net interchange download-2

By passing the other kwarg, I can compare two different datasets. Here I am comparing the raw dataset to the final dataset. Note that this function can also be used to compare intermediate datasets in the gridemissions workflow. This can get quite messy, so I restricted the x-axis to 2021 here. The red line for "after" is now at 0 -- which is the expected result from the data reconciliation algorithm. download-3

Plot adjustments.ipynb

A dropdown can be used to select any timeseries and see how it was adjusted. For example: BPAT hydro.

download-1

Commit log

commit 94cba9ba2a41c200908956a358bdd0a4710b87f3
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 17:28:58 2024 -0700

    ge_download --bulk

commit 86651c91b4690ad257a85381f419ee8ad6a64740
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 17:10:02 2024 -0700

    notebook reorg

commit 00bf4361542f6a7fd96951e17e96a9a2facee3a3
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 17:09:29 2024 -0700

    cleaning plot notebook

commit d98db0868b9b2207d0ecb8a461da75a6098c0469
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 15:59:31 2024 -0700

    new function load_bulk

commit 9cd1f8e88578ce528b888a7a21e11630126d451b
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 15:29:45 2024 -0700

    formatting for `ge_update --help`

commit fb310d2b7de8eb1bfc7b00c68956f8553042e576
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Tue Jun 11 15:29:27 2024 -0700

    update to APEN paper figures for analysis of adjustments

commit 067e200e96d84d6c168123e5da7dbde916b48618
Author: Jacques de Chalendar <jdechalendar@stanford.edu>
Date:   Fri Jun 7 14:23:38 2024 -0700

    updates to viz.reports

    * summ_stats made compatible with more recent versions of
      pandas/matplotlib
    * annual_plot_weekly minor bug fixes
    * heatmap_report updated to use GraphData
    * updated units
    * report script
      * removed --year flag, running all years instead for --report heatmap
      * updated to use GraphData
      * updated to use bulk dataset
    * bulk data demo notebook: prioritize later timesteps when merging six-month files
jdechalendar commented 5 months ago

@zaneselvans I think the plots here can be used to provide some answers to questions you had asked about how much of an adjustment was being made by the different data processing tools in this repo, especially the data-reconciliation step.