Global energy storage database integration and storage workflow updates

[maartenbrinkerink]

Description

1. Created an individual folder osemosys_global/storage for all storage related scripts and created an individual storage rule within preprocess.py. The storage rule comes after the powerplant and transmission rules either appending data to already existing parameters (e.g. ResidualCapacity.csv) or by creating new parameter files (e.g. CapitalCostStorage.csv).
2. Created the storage_parameters entry in the config file that defines baseline storage parameters per storage technology. Storage technologies can be fully user defined.
3. Created the user_defined_capacity_storage entry in the config file that allows for (multiple) user defined entries per storage technology. This also allows for multiple residual capacities per technology at different time intervals. Other parameters (e.g. capital costs) per user defined technology can only be defined once.
4. Enabled year, region and technology specific storage build rate constraints within resources/data/storage_build_rates.csv following #222. This now exists alongside similar build rate constraints for generators (powerplant_build_rates.csv) and transmission (transmission_build_rates.csv).
5. Integrated the GESDB database in the workflow that defines ResidualCapacity and ResidualStorageCapacity. The general steps are; a: Capacity (GW) and storage capacity (GWh) is pulled from the database at plant level (+-2500 entries currently) and grouped to OG storage technologies as defined within storage/constants.py. b: Different 'Status' levels are grouped within storage/constants.py as a basis to be able to define currently existing, planned and already retired storage technologies. Relevant constants are INACTIVE_VARS, ACTIVE_VARS, NEW_VARS. c: a and b, together with default operational life data (defined in the storage_parameters entry in the config file) and assumed build/retirement years for entries where this data does not exist within the GESDB database (BUILD_YEAR and RETIREMENT_YEAR within storage/constants.py), allows us to populate ResidualCapacity.csv and ResidualStorageCapacity.csv. d: Note that ResidualCapacity.csv and ResidualStorageCapacity.csv are linearly linked. I.e. ResidualStorageCapacity.csv (PJ) = ResidualCapacity.csv (GW) * storage duration (hours) / 277.777778 (converstion to PJ). Storage durations per entry are either derived from the GESDB dataset in case the DURATION_TYPE constant in storage/constants.py is set to Historical or set to default values if DURATION_TYPE is set to Default. e: A current limitation in the dataset integration is that the spatial mapping occurs based on state/province --> OG node (see resources/data/GESDB_region_mapping.csv) which is predominantly an issue for the U.S. nodes as data within one state needs to be associated with one OG node. U.S. states are often split across multiple OG nodes. As discussed before, in the future we should move to using shapefiles to assign individual plant data (for generators and storage) to OG nodes.

Issue Ticket Number

175, #222, #187, #67

Questions/discussion points

There are a number of questions/discussion points that I wanted to raise. Feedback is appreciated!

1. The workflow right now is not yet functional as an 'out of domain' error pops up without a clear reason for it occurring.

The storage techs are both defined in TECHNOLOGY.csv as well as STORAGE.csv and TechnologyFromStorage and TechnologyToStorage are defined similarly compared to master.

@trevorb1 reckons it might have to do with the bottom line in the below screenshot from osemosys_fast_preprocessed.txt but no certainty there either. @tniet or @abhishek0208 if either of you could run the workflow from the PR and see if you can pinpoint the issue that would be appreciated.

2. Please have a look at the attached Excel file and let me know if the applied units for storage make sense. Units on the left are the input units for users and units on the right are the outputs from the scripts (i.e. the units that go into the parameter csv's). Units overview.xlsx
3. Assuming the above units are correct, I applied CapacityToActivityUnit for the PWR objects only (e.g. PWRSDSINDEA01) to convert values from ResidualCapacity.csv and have set values for ResidualStorageCapacity.csv in PJ. Is this correct?
4. I've set capital costs to CapitalCostStorage.csv to the storage object (e.g. SDSINDEA01) and FixedCost and VariableCost to the PWR object (e.g. PWRSDSINDEA01) as these parameters do not exist for storage objects I believe. Is this setup correct?
5. If it is, I assume this means that values for FixedCost should be in m$/GW whereas for CapitalCostStorage and VariableCost it is in m$/PJ?
6. Continuing that logic, let's assume that we use a 4-hour battery storage object for the cost basis (ATB 2024) with a total CAPEX of 1938m$/GW and fixed annual cost of 44m$/GW. Assuming we define CapitalCostStorage I assumed the correct value would be 1938m$/GW / 4 hours = 484.5m$/GWh (followed by conversion to PJ). Assuming we define FixedCost I assumed the correct value would be 44m$/GW / 4 hours = 11m$/GW. The latter is assuming we cannot make the correlation within OSeMOSYS that 1 GW of PWR capacity equals 4 GWh of Storage capacity (to be confirmed).
7. For VariableCost, should the costs be assigned to only one mode of operation (assuming this represents charge/discharge) or both? Currently only set to mode 2 as this is where the roundtrip losses are defined (in OAR).
8. TotalAnnualMinStorageCapacityInvestment does not exist? We set user defined planned PWR capacities to TotalAnnualMinCapacityInvestment (also for the PWR object associated to storage) but what about the planned storage capacities? Is there any point defining it for the PWR object if we can't do it for the storage object?
9. As mentioned before there is a linear link between ResidualCapacity and ResidualStorageCapacity. I.e. we define storage durations, either through default values (e.g. 4 hours for SDS) or derived from the GESDB dataset, and multiply GW values from the GESDB dataset (ResidualCapacity) to derive GWh/PJ values (ResidualStorageCapacity). Does this link make sense? Related to the Charge/DischargeRate point below, is there any way to create a clear link between the power tech (e.g. PWRSDSINDEA01) and storage tech (and SDSINDEA01) that constrains the hourly use in a scalable fashion? I.e. in a way that it in/decreases by year depending on residual capacity as well as uptake of new capacities.
10. Since residual capacities change by year, and it seems that StorageMaxChargeRate[r,s] and StorageMaxDischargeRate[r,s] (link) cannot be defined by year, I'm unsure what to use for these parameters (now uses default value of 99999). Furthermore, how can charge rates be scaled for new capacities (both for storage techs that have residual capacity, as well as new techs)?
11. ~Generating S3_NetChargeWithinYear... and Generating S4_NetChargeWithinDay... Steps take incredibly long. Perhaps because charge rates are not defined?~ Solved with #463c9e4

Happy to chat about this tomorrow as it is quite a bit to take in:)