feat: integrate DOE flexibility data and percentage-based input

[dongqi-wu]

Pull Request doc

Purpose

Integrate pre-processed EIA-ID based flexibility data from DOE flexibility data into REISE modelling. The DOE data will be downloaded from blob storage first if not found in the case folder. The EIA ID of buses is stored in column 18 of the bus field in case.mat. This file is manually modified for now, but in the future this info will be integrated into powersimdata.

What the code is doing

Read the percentage based LSE flexibility, match with load buses and convert to MW amount

Testing

run_scenario_gurobi() using TX sub-system data

Where to look

read.jl: additional logic for reading DOE data model.jl: convert percentage to MW

Time estimate

15 min

[dongqi-wu]

Most of the comments have been addressed in the recent commit. This version lets the user to choose flexibility input between using DOE data (downloaded from the BLOB storage) or local csv files. The code assumes the EIA ID of bus (feat: EIA ID of buses in the synthetic grid #623) is present in the .mat case file.

[dongqi-wu]

I think this mostly looks good to me. Please see my comments below.

Additionally, do we have a plan for mixing user-provided profiles (i.e., the original method) and the profiles derived from EIA data? I remember we talked about it before, but it's been a while.

The plan was to make the model take both, with the csv files overriding the EIA ones if needed. This will be the next version.

[dongqi-wu]

I just pushed another commit to enable using both DOE and csv flexibility, while csv columns overriding the defaults ones from DOE data. This requires modifying the demand_flexibility.cost_up/dn after reading flex_cost_up/dn.csv into the demand_flexibility struct, as the total number of flexible bus will be different than the number of columns in the csvs when DOE data is used, and the dimension of the DF need to be modified accordingly. There might be cleaner ways but for this version I just want the logic to be reviewed.

Also the test shows

ERROR: Could not find either git-tree-sha1 or path for package DelimitedFiles

idk if its my problem or something with the pip equivalent for Julia, please let me know if you have seen this before.

[lanesmith]

I just pushed another commit to enable using both DOE and csv flexibility, while csv columns overriding the defaults ones from DOE data. This requires modifying the demand_flexibility.cost_up/dn after reading flex_cost_up/dn.csv into the demand_flexibility struct, as the total number of flexible bus will be different than the number of columns in the csvs when DOE data is used, and the dimension of the DF need to be modified accordingly. There might be cleaner ways but for this version I just want the logic to be reviewed.

Thanks! I'll be sure to check it out.

Also the test shows

ERROR: Could not find either git-tree-sha1 or path for package DelimitedFiles

idk if its my problem or something with the pip equivalent for Julia, please let me know if you have seen this before.

The lint test is indicating that model.jl and read.jl need to be formatted. Don't worry about the Julia nightly test for the time being as it is not an issue on your end. The Julia nightly test is not a required test for the PR to be merged.

[lanesmith]

We should probably also add a warning (not an error) in read.jl's read_demand_flexibility in the event that DOE flexibility is enabled and demand flexibility profiles are provided. This warning would let the user know the policy for taking the provided demand flexibility profiles over the DOE flexibility profiles.

[lanesmith]

@dongqi-wu, thank you for your latest commit! For the most part, I think this is good to go on my end. Please be sure to address my unresolved comment about formatting and my comment about adding a warning above. Additionally, I'm not sure what happened, but it seems that you already added in some of the latest changes from the develop branch (at least from what I can tell by the latest commit's changes to read.jl). Perhaps this will be resolved when you rebase.

Once the above is addressed and the code is rebased, this should be able to be approved on my end.

[dongqi-wu]

@dongqi-wu, thank you for your latest commit! For the most part, I think this is good to go on my end. Please be sure to address my unresolved comment about formatting and my comment about adding a warning above. Additionally, I'm not sure what happened, but it seems that you already added in some of the latest changes from the develop branch (at least from what I can tell by the latest commit's changes to read.jl). Perhaps this will be resolved when you rebase.

Once the above is addressed and the code is rebased, this should be able to be approved on my end.

Sure I will check the comments again and run juliaformatter before trying to rebase. Regarding the latest changes, I tried to copy them into read.jl before the last commit to see if the conflict still appears.

[rouille]

@jenhagg, we will need to include an extra column for the bus data frame when we export the grid.pkl to CSVs. Just a reminder that we need to work on it

[lanesmith]

@jenhagg, we will need to include an extra column for the bus data frame when we export the grid.pkl to CSVs. Just a reminder that we need to work on it

It's been a while, but I just want to link PowerSimData PR #623, which specifies this feature, albeit under the old convention.

[dongqi-wu]

Looks good to me. Once @BainanXia's comments are addressed, this should be good to go. Please also make sure that any tests you have are able to work now that your branch is rebased.

sure I will fix them soon.

[dongqi-wu]

Looks good to me. Once @BainanXia's comments are addressed, this should be good to go. Please also make sure that any tests you have are able to work now that your branch is rebased.

for the testing, I just found the refactored code give the error below using the old input files. was the file changed in the past?

# Generator costs

gencost = DataFrames.DataFrame(CSV.File(joinpath(filepath, "gencost.csv"))) case["gencost"] = convert(Matrix{Float64}, gencost)

[lanesmith]

for the testing, I just found the refactored code give the error below using the old input files. was the file changed in the past?

# Generator costs

gencost = DataFrames.DataFrame(CSV.File(joinpath(filepath, "gencost.csv"))) case["gencost"] = convert(Matrix{Float64}, gencost)

Yeah, the change to REISE.jl that you rebased changed how we consider input files. Now, instead of using the case.mat files, we output grid information from PowerSimData as a .pkl file. If you use the Python wrapper, the .pkl is taken as input and is used to produce the different .csv files that you now see read in in read.jl. If you use the Julia interface, you need to have those .csv files. We can get you new test files for the the Texas grid, though you'll need to add in the column of EIA flexibility information to the appropriate .csv file.

[dongqi-wu]

for the testing, I just found the refactored code give the error below using the old input files. was the file changed in the past?

# Generator costs

gencost = DataFrames.DataFrame(CSV.File(joinpath(filepath, "gencost.csv"))) case["gencost"] = convert(Matrix{Float64}, gencost)

Yeah, the change to REISE.jl that you rebased changed how we consider input files. Now, instead of using the case.mat files, we output grid information from PowerSimData as a .pkl file. If you use the Python wrapper, the .pkl is taken as input and is used to produce the different .csv files that you now see read in in read.jl. If you use the Julia interface, you need to have those .csv files. We can get you new test files for the the Texas grid, though you'll need to add in the column of EIA flexibility information to the appropriate .csv file.

Thank you. How do I get the new files? I can run my local script to get the EIA ID column myself.

[lanesmith]

Thank you. How do I get the new files? I can run my local script to get the EIA ID column myself.

Sorry I didn't share it yesterday. I thought you said you had the .csv files. Using PowerSimData, you can create the .pkl file as follows:

import os
import pickle
import powersimdata
from powersimdata import Scenario

# Create grid object
scenario = Scenario()
scenario.set_grid(grid_model="usa_tamu", interconnect="Texas")
grid = scenario.get_grid()

# Dump in a .pkl file
filepath = INSERT FILE PATH STRING HERE
with open(os.path.join(filepath, "grid.pkl"), "wb") as f:
    pickle.dump(grid, f, protocol=pickle.HIGHEST_PROTOCOL)

Once you have the .pkl file, you should be able to run REISE.jl using the Python interface, as is specified here. If you just want to run REISE.jl using the Julia interface, you will need to run the appropriate functions in pyreisejl/utility/call.py to create the .csv files from the .pkl file. This latter path might be preferred so you can add the EIA ID column.

[dongqi-wu]

Thank you. How do I get the new files? I can run my local script to get the EIA ID column myself.

Sorry I didn't share it yesterday. I thought you said you had the .csv files. Using PowerSimData, you can create the .pkl file as follows:

import os
import pickle
import powersimdata
from powersimdata import Scenario

# Create grid object
scenario = Scenario()
scenario.set_grid(grid_model="usa_tamu", interconnect="Texas")
grid = scenario.get_grid()

# Dump in a .pkl file
filepath = INSERT FILE PATH STRING HERE
with open(os.path.join(filepath, "grid.pkl"), "wb") as f:
    pickle.dump(grid, f, protocol=pickle.HIGHEST_PROTOCOL)

Once you have the .pkl file, you should be able to run REISE.jl using the Python interface, as is specified here. If you just want to run REISE.jl using the Julia interface, you will need to run the appropriate functions in pyreisejl/utility/call.py to create the .csv files from the .pkl file. This latter path might be preferred so you can add the EIA ID column.

Thanks!

[dongqi-wu]

Thank you. How do I get the new files? I can run my local script to get the EIA ID column myself.

Sorry I didn't share it yesterday. I thought you said you had the .csv files. Using PowerSimData, you can create the .pkl file as follows:

import os
import pickle
import powersimdata
from powersimdata import Scenario

# Create grid object
scenario = Scenario()
scenario.set_grid(grid_model="usa_tamu", interconnect="Texas")
grid = scenario.get_grid()

# Dump in a .pkl file
filepath = INSERT FILE PATH STRING HERE
with open(os.path.join(filepath, "grid.pkl"), "wb") as f:
    pickle.dump(grid, f, protocol=pickle.HIGHEST_PROTOCOL)

Once you have the .pkl file, you should be able to run REISE.jl using the Python interface, as is specified here. If you just want to run REISE.jl using the Julia interface, you will need to run the appropriate functions in pyreisejl/utility/call.py to create the .csv files from the .pkl file. This latter path might be preferred so you can add the EIA ID column.

@lanesmith any idea how the gencost functions in the previous update works? the Python function only produces one gencost.csv from the pickle file, but it seems the read.jl is taking two at a time now.

edit: the previous read gencost.csv line gave an error as well about converting Dataframe to Matrix. The current version is after rebasing on top of the newest commit, but the code is not tested due to the input file mismatch.

[lanesmith]

@lanesmith any idea how the gencost functions in the previous update works? the Python function only produces one gencost.csv from the pickle file, but it seems the read.jl is taking two at a time now.

edit: the previous read gencost.csv line gave an error as well about converting Dataframe to Matrix. The current version is after rebasing on top of the newest commit, but the code is not tested due to the input file mismatch.

Hi @dongqi-wu, sorry I'm just now getting back to you. We recently updated REISE.jl and PowerSimData so that some of the processes we had to linearize gencosts and update pmins and ramp rates were moved from REISE.jl to PowerSimData. You can now run the following to create the necessary .pkl file (i.e., will include appropriate grid.gencost["after"] and updated pmin and ramp_30 values that will reflect what REISE.jl used to implement):

import os
import pickle
import powersimdata
from powersimdata import Scenario
from powersimdata.input.configure import linearize_gencost, adjust_pmin, adjust_ramp30

# Create grid object
scenario = Scenario()
scenario.set_grid(grid_model="usa_tamu", interconnect="Texas")
grid = scenario.get_grid()
adjust_pmin(grid)
adjust_ramp30(grid.plant)
grid.gencost["after"] = linearize_gencost(grid.gencost["before"], grid.plant, num_segments=1)

# Dump in a .pkl file
filepath = INSERT FILE PATH STRING HERE
with open(os.path.join(filepath, "grid.pkl"), "wb") as f:
    pickle.dump(grid, f, protocol=pickle.HIGHEST_PROTOCOL)

Be sure that you've got the latest version of PowerSimData's develop branch before you do this. Once you have the new .pkl file, you should be able to run REISE.jl's Python interface as specified before. This should take care of creating those different gencost .csv files (as well as the other .csv files) you were asking about.

[dongqi-wu]

@lanesmith any idea how the gencost functions in the previous update works? the Python function only produces one gencost.csv from the pickle file, but it seems the read.jl is taking two at a time now. edit: the previous read gencost.csv line gave an error as well about converting Dataframe to Matrix. The current version is after rebasing on top of the newest commit, but the code is not tested due to the input file mismatch.

Hi @dongqi-wu, sorry I'm just now getting back to you. We recently updated REISE.jl and PowerSimData so that some of the processes we had to linearize gencosts and update pmins and ramp rates were moved from REISE.jl to PowerSimData. You can now run the following to create the necessary .pkl file (i.e., will include appropriate grid.gencost["after"] and updated pmin and ramp_30 values that will reflect what REISE.jl used to implement):

import os
import pickle
import powersimdata
from powersimdata import Scenario
from powersimdata.input.configure import linearize_gencost, adjust_pmin, adjust_ramp30

# Create grid object
scenario = Scenario()
scenario.set_grid(grid_model="usa_tamu", interconnect="Texas")
grid = scenario.get_grid()
adjust_pmin(grid)
adjust_ramp30(grid.plant)
grid.gencost["after"] = linearize_gencost(grid.gencost["before"], grid.plant, num_segments=1)

# Dump in a .pkl file
filepath = INSERT FILE PATH STRING HERE
with open(os.path.join(filepath, "grid.pkl"), "wb") as f:
    pickle.dump(grid, f, protocol=pickle.HIGHEST_PROTOCOL)

Be sure that you've got the latest version of PowerSimData's develop branch before you do this. Once you have the new .pkl file, you should be able to run REISE.jl's Python interface as specified before. This should take care of creating those different gencost .csv files (as well as the other .csv files) you were asking about.

Thanks, I will try and get back to you.

edit: I have tested the new input files and they all work (with or without DOE/CSV data). Please see the newest commit, it should also pass the style check as well.

[lanesmith]

Thanks, I will try and get back to you.

edit: I have tested the new input files and they all work (with or without DOE/CSV data). Please see the newest commit, it should also pass the style check as well.

Thanks for verifying that, @dongqi-wu. I've looked over the latest commit and it looks good to me. So long as the checks pass, I think this is good to merge. @BainanXia, do you have any further feedback? Also, @BainanXia, I am not able to run the above checks, can you allow them to run?

[BainanXia]

Thanks, I will try and get back to you. edit: I have tested the new input files and they all work (with or without DOE/CSV data). Please see the newest commit, it should also pass the style check as well.

Thanks for verifying that, @dongqi-wu. I've looked over the latest commit and it looks good to me. So long as the checks pass, I think this is good to merge. @BainanXia, do you have any further feedback? Also, @BainanXia, I am not able to run the above checks, can you allow them to run?

It looks good to me as well and I can't run the checks either. It says "Waiting for status to be reported". Not sure what's going on here. Maybe @jenhagg has a better idea?

[jenhagg]

Thanks, I will try and get back to you. edit: I have tested the new input files and they all work (with or without DOE/CSV data). Please see the newest commit, it should also pass the style check as well.

Thanks for verifying that, @dongqi-wu. I've looked over the latest commit and it looks good to me. So long as the checks pass, I think this is good to merge. @BainanXia, do you have any further feedback? Also, @BainanXia, I am not able to run the above checks, can you allow them to run?

It looks good to me as well and I can't run the checks either. It says "Waiting for status to be reported". Not sure what's going on here. Maybe @jenhagg has a better idea?

Not sure why they wouldn't be triggered other than github being unreliable. Two options I can think of are pushing a new commit, or if that doesn't work, adding a workflow_dispatch trigger to the checks so we can run them manually.