Manage capacity expansion model (CEM) simulation data

[danielolsen]

:rocket:

• [x] Is your feature request essential for your project?

Describe the workflow you want to enable

I wish I could customize a capacity expansion model (CEM) simulation, send the prepared information to a remote server (or container) where it will be solved, and then have all relevant inputs and outputs be accessible to everyone with server access through a simple user interface. There three tasks roughly correspond to the existing Create, Execute, and Analyze states of the existing Scenario object, which is designed to manage the information for a production cost model (PCM). The interface for managing and running CEM simulations could be an analogous user-facing class, containing child objects which perform more specific tasks (these could also be analogous to the existing Scenario child classes).

Describe your proposed implementation

Customization includes:

• Modifications to the 'base grid' to represent the currently-built grid infrastructure:
  • the existing ChangeTable methods modify what's present as a starting point
  • existing infrastructure could also be marked for retirement at a future date
• Definitions of potential new infrastructure:
  • Baseline expansion candidates: generation, transmission, and storage. Future expansion candidates: hydrogen and CCUS infrastructure
  • Generator expansion information:
  • Where (bus or substation ID)
  • Generator type
  • Cost per MW of new capacity
  • Limit of new capacity at this bus (potentially unlimited)
  • Operating costs (which might need to be specified differently than currently in gencost, since we don't know the capacity of the generator, so the x-axis of the curve can't be MW of generation)
  • Transmission expansion information:
  • to and from buses (or substations?)
  • Voltage (if the branch is defined by substations, not buses. Otherwise the buses should have a common defined voltage)
  • Cost per MW of new capacity
  • maximum potential capacity (potentially unlimited)
  • some way of determining the line impedance (not sure how we want to do this though, since greater capacity lines will tend to have lower impedance).
  • Note: this specification assumes that transmission line construction on each path is a continuous variable (e.g. build a transmission line of 0-1000MW) rather than discrete (e.g. either build a 100 MW line, or a 500 MW line, or a 1000 MW line). Discrete variables are closer to real-world decision-making, and allow each potential project to have its impedance and capacity set exogenously, but transform the optimization problem from linear to mixed-integer, which increases the computational complexity.
  • Storage expansion information:
  • Where (bus or substation ID)
  • a cost per MW of new power capacity
  • a cost per MW of new energy capacity
  • a limit of how much new power capacity can be built at this bus (potentially unlimited)
  • a limit of how much new energy capacity can be built at this bus (potentially unlimited)
  • Constraints around the power:energy ratio (e.g. assume 4-hr storage, or something between 2-hr and 8-hr)
  • Charging and discharging efficiency
  • Note: default per-MW costs for generation, transmission, and storage can be derived from the powersimdata.design.investment module
  • If information on land uses is available, this information may be used to modify the expansion candidates, e.g. setting the availability of wind plant expansion to 0 throughout sensitive areas for migratory birds.
• CEM Scenario meta-information:
  • Spatial clustering information: either a mapping of buses to clusters (i.e. the busmap input to pypsa.networkclustering.get_clustering_from_busmap) or the appropriate parameters to generate the same via other algorithms.
  • Temporal clustering information: either a mapping of our starting 8760 timestamps to representative snapshots/timeslices (to be ingested by some downstream function for aggregation) or the appropriate parameters to generate the same via other algorithms.
  • Multiple investment period information: see the PyPSA example: https://pypsa.readthedocs.io/en/latest/examples/multi-investment-optimisation.html#Multi-Investment-Optimization
  • In addition to the multiple investment periods, we may have additional non-investment operational periods, e.g. investments can be made in {2024, 2026, 2028, 2030} but operations are also considered in {2023, 2025, 2027, 2029}. The operational costs in each year will need to be weighted somehow against capital investments, but this could be calculated somewhat automatically using assumptions about financing costs, discount rates, whether there's a 'terminal' year that represents all future years, etc.
  • 'Global' constraints that are already built into PyPSA or other CEM tools: e.g. a limit on the total CO2 for a given year, see https://pypsa.readthedocs.io/en/latest/components.html#global-constraints
  • 'Global' constraints that aren't already built into PyPSA or other CEM tools: e.g. a limit on the total amount of a certain resource that can be installed in a given year (to represent manufacturing capacity or labor constraints).

Once customization is complete, preparing the information would generate a unique ID for this simulation, apply the specified customizations and then create the required input files that can be run with a CEM tool and upload them to the appropriate place on the remote server. For PyPSA, this could be a pickled Network object; for Switch, this could be a set of CSVs; etc. The user would then be able to launch the CEM process on the server, specifying any additional information about the solution process (e.g. the solver to be used, the number of threads, etc.). There may also be a post-simulation step to transform the results into a format that's more standardized, to be able to more easily retrieve data from simulations which may have been run using different CEM tools. These 'launching' and 'extraction' functions may live in separate repositories which are thin wrappers around each CEM tool, e.g. SwitchWrapper, PyPSAWrapper, etc.

Once the simulation is complete, the user would be able to instantiate an object using the Scenario ID, and this object would be an interface to all relevant input and output information (retrieving each from the local machine or transferring from the remote server as applicable). This information would include:

• The starting grid
• Investments and retirements for each investment period
  • Analogously, Grid objects representing the infrastructure in a given year
• Time-series information for each operational year simulated
• Generator dispatch and power flow for each operational year simulated

Relevant sub-tasks

• 638

• 641

This list may be incomplete.

[danielolsen]

Here's a crude diagram of how we might enable user input in the Create state (blue boxes) as well as process the data during Execute state to produce a Network object (orange boxes). I've broken up the three top-level bullets for customization into three different data structures/objects, but this is only one potential design and we may want to rethink the user interface and the internal data storage.