Suggestion for Demand Module Refactor

[trevorb1]

Changes proposed in this Pull Request

Description

Suggestion to implement a strategy design to work with demand data. This is hopfully a little more extensible for when new demand sources are added for any sector. Im not saying we merge this, just meant to get a conversation going on how to deal with new datasets as they pop up!

In this example I only implement the reading and writing of EIA data (where writing is really just parsing). I think the reading is fairly clean, however, the writing probably needs a little work. I am not super familiar with how to dissagregation happens between different sources. So maybe instead of of a WriteEia strategy, it should be more along the lines of dissagregation strategies (based on BA, states, regions, population, ect...) or by sector. @ktehranchi may have better insights here!

Or maybe this just over complicates demand. haha. It would just be nice to reuse the demand module to pull heat demand (and eventually transport demand) data, which I think this module should allow.

Adding New Data

To read in a new dataset, the user will:

1. Add a strategy that inherits from ReadStrategy class (ie. ReadEfs)
2. Populate methods for _read_data(...) and _format_data(...) for the new class
3. The _format_data(...) method must output long formatted data (with column labels of HOUR, REGION, YEAR, FUEL, SECTOR. However, format and labels are def up for discussion!

To write out a new dataset, the user will:

1. Add a strategy that inherits from WriteStrategy class (ie. WriteEfs) -- As noted above, this should maybe be changed to dissagregation methods rather than tied to datasets.
2. Populate the update_load_dissagregation_names(..) and get_demand_buses(..) methods

Example

n = pypsa.Network(snakemake.input.base_network)

demand_converter = Context(ReadEia(), WriteEia())

# optional arguments of 'fuel', 'sector', 'year'
demand = demand_converter.retrieve_demand("eia_data.csv", n)

Other Notes

Pros of this method are:

• I think it will allow us to slot in new data sets as they come up easily
• Allows for reuse of code for sector demand
• Can lead itself easily to updating to xarray processing (Im looking at you EFS!) haha :)

Cons of this method are:

• There is overhead in getting this up and running (especially the writing may need a little more thought)
• May be overkill for our needs
• There may actually be a better design pattern that I just dont know about!

Checklist

• [ ] I tested my contribution locally and it seems to work fine.
• [ ] Code and workflow changes are sufficiently documented.
• [ ] Changed dependencies are added to envs/environment.yaml.
• [ ] Changes in configuration options are added in all of config.default.yaml.
• [ ] Changes in configuration options are also documented in doc/configtables/*.csv.

[trevorb1]

Chatting with @ktehranchi, likely better to turn the Write strategies into dissagregation strategies instead, then just pass in any renaming dictionaries as kwargs

[trevorb1]

We should revisist this now that PR #247 has been merged, but not applied

[trevorb1]

Edit, Found a bug. Not done yet

Hi! I have finished this PR now, I believe. The major update (as described in the original description) is the introduction of a strategy design to handle the reading of different datasets, and the writing via different dissagregation strategies.

The high level UML is given below. Anything in color is implemented, while the white boxes show what I plan to work on next (note, WritePopulation implements breakthrough energy population, rather than county level population).

Other notable changes in this PR are:

• EFS data has been updated to retain all info upon reading (rather than aggregate eveything to state level). This will be benifical for rolling horizon / myopic studies and/or tracking sector electrical demand
• Some logic was updated to reflect the handling of larger datasets (for example, the timezone adjustments got rid of the utcoffset() call, and parsing of EFS was changed to vectorized operations). End results are the same, just processing logic is different.
• Growth rates from EFS and the AEO are added. The AEO only gives end use energy, rather than at a per-fuel level. Note, while logic has been added to extract this data, applying the growth rates have not been done.
• Removal of ADS from build_demand
• You must put a planning horizon in now. ie. if you want to run validation, you need to put 2019 for example.

Finally, the config file has been updated to reflect these changes with the following for electrical demand:

  demand: 
    profile: efs # efs, eia
    scale: 1 # efs, aeo, or a number 
    disaggregation: pop # pop
    scenario: 
      efs_case: reference # reference, medium, high
      efs_speed: moderate # slow, moderate, rapid

and the following for sectors

# docs :
sector:
  demand:
    profile:
      residential: eulp # efs, eulp
      commercial: eulp # efs, eulp
      transport: efs # efs
      industry: efs # efs
    scale:
      residential: aeo # efs, aeo
      commercial: aeo # efs, aeo
      transport: aeo # efs, aeo
      industry: aeo # efs, aeo
    disaggregation: 
      residential: pop # pop
      commercial: pop # pop
      transport: pop # pop
      industry: pop # pop
    scenarios:
      aeo: reference

[trevorb1]

Okay great, this should be good to merge. @ktehranchi can you give it a quick run on your side though, please. Make sure you grab the config file demand updates! I have checked that both EFS and EIA data produce the same results from before this PR.

For future reference, I will just leave what I see the pros and cons of this method (expanded from initial PR description):

• Pros of this method are:

  • New data sets can very easily be added. Moreover, we can parse datasets for different purposes (ie. EFS for electricity only, and also sector coupling) consistently
  • Dissagregation strategies can be added and compared eaisly, as only a new LAF series needs to be generated. We can be confident the rest of the code is the same
  • For myopic/rolling horizon studies, this implementation will be much easier to integrate then the old one
  • As demand is one of the critical factors to ensure is correct for cost optimizations, taking the time to set up code that can (in theory) be tested easily is good.

• Cons of this method are:

  • The ReadStrategy retains as much data as possible. If you are running only a single year for a single sector, you may be processing a considerable amount of extra data before filtering it out. (Although, xarray will solve much of this)
  • There is overhead with implementing this. This does three things; 1) it may make it less accessible for new users to understand what is happening, and 2) it may be overkill for our needs 3) requires a bit of time to first understand the control flow before troubleshooting
  • There may actually be a better design pattern to implement that I just dont know about!