Re paper - Githubissues

dguittet commented 1 year ago

Addresses issue:

Scripts for running simulations for the report:

Wind + PEM case:

dispatches/case_studies/renewables_case/run_pricetaker_wind_PEM.py: Wind + PEM Price-taker Design Optimization
dispatches/case_studies/renewables_case/RE_surrogate_optimization_steadystate.py: Market Surrogate Design Optimization
dispatches/case_studies/renewables_case/run_double_loop_PEM.py: Wind + PEM Double Loop Simulation for Validation
dispatches/case_studies/renewables_case/SurrogateDesignResults.ipynb: Market Surrogate Design and Validation Plotting

Wind + Battery case:

dispatches/case_studies/renewables_case/run_pricetaker_battery_ratio_size.py: Wind + Battery Price-taker Design Optimization
dispatches/case_studies/renewables_case/run_double_loop_battery.py: Wind + Battery Double Loop Simulation
dispatches/case_studies/renewables_case/run_double_loop_battery_parametrized.py: Wind + Battery with Parametrized Bidder Double Loop Simulation

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the license terms described in the LICENSE.md and COPYRIGHT.md file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 91.20% and project coverage change: -0.05% :warning:

Comparison is base (f4d811b) 89.87% compared to head (1b1c6bf) 89.83%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #208 +/- ## ========================================== - Coverage 89.87% 89.83% -0.05% ========================================== Files 77 77 Lines 9257 9307 +50 ========================================== + Hits 8320 8361 +41 - Misses 937 946 +9 ``` | [Files Changed](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches) | Coverage Δ | | |---|---|---| | [...studies/renewables\_case/PEM\_parametrized\_bidder.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL1BFTV9wYXJhbWV0cml6ZWRfYmlkZGVyLnB5) | `100.00% <ø> (ø)` | | | [.../tests/test\_multiperiod\_wind\_battery\_doubleloop.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3Rlc3RzL3Rlc3RfbXVsdGlwZXJpb2Rfd2luZF9iYXR0ZXJ5X2RvdWJsZWxvb3AucHk=) | `100.00% <ø> (ø)` | | | [...tudies/renewables\_case/wind\_battery\_double\_loop.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3dpbmRfYmF0dGVyeV9kb3VibGVfbG9vcC5weQ==) | `77.86% <0.00%> (-1.47%)` | :arrow_down: | | [...se\_studies/renewables\_case/wind\_PEM\_double\_loop.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3dpbmRfUEVNX2RvdWJsZV9sb29wLnB5) | `92.10% <50.00%> (-0.76%)` | :arrow_down: | | [...bles\_case/RE\_surrogate\_optimization\_steadystate.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL1JFX3N1cnJvZ2F0ZV9vcHRpbWl6YXRpb25fc3RlYWR5c3RhdGUucHk=) | `89.65% <70.00%> (-1.86%)` | :arrow_down: | | [...es/case\_studies/renewables\_case/load\_parameters.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL2xvYWRfcGFyYW1ldGVycy5weQ==) | `97.18% <92.85%> (-2.82%)` | :arrow_down: | | [...tches/case\_studies/renewables\_case/RE\_flowsheet.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL1JFX2Zsb3dzaGVldC5weQ==) | `95.89% <100.00%> (ø)` | | | [...studies/renewables\_case/tests/test\_RE\_flowsheet.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3Rlc3RzL3Rlc3RfUkVfZmxvd3NoZWV0LnB5) | `96.61% <100.00%> (+0.27%)` | :arrow_up: | | [...newables\_case/tests/test\_solar\_battery\_hydrogen.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3Rlc3RzL3Rlc3Rfc29sYXJfYmF0dGVyeV9oeWRyb2dlbi5weQ==) | `100.00% <100.00%> (ø)` | | | [...tudies/renewables\_case/tests/test\_surrogate\_opt.py](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches#diff-ZGlzcGF0Y2hlcy9jYXNlX3N0dWRpZXMvcmVuZXdhYmxlc19jYXNlL3Rlc3RzL3Rlc3Rfc3Vycm9nYXRlX29wdC5weQ==) | `72.72% <100.00%> (-1.19%)` | :arrow_down: | | ... and [4 more](https://app.codecov.io/gh/gmlc-dispatches/dispatches/pull/208?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=gmlc-dispatches) | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dguittet commented 1 year ago

@jsiirola @lbianchi-lbl The "Checks / Tests (non-editable install)" part of CI is failing because the solver objective (NPV in this case) on Windows is significantly worse than that on Linux:

FAILED case_studies/renewables_case/tests/test_RE_flowsheet.py::test_wind_battery_pem_tank_turb_optimize_simple - assert 967121084.0761821 == 1018975372 � 1.0e+07
  comparison failed
  Obtained: 967121084.0761821
  Expected: 1018975372 � 1.0e+07

Not sure why the "Checks / Test" part of CI is passing with the Windows solver finding the same optimal point.

lbianchi-lbl commented 1 year ago

@jsiirola @lbianchi-lbl The "Checks / Tests (non-editable install)" part of CI is failing because the solver objective (NPV in this case) on Windows is significantly worse than that on Linux:
FAILED case_studies/renewables_case/tests/test_RE_flowsheet.py::test_wind_battery_pem_tank_turb_optimize_simple - assert 967121084.0761821 == 1018975372 � 1.0e+07
  comparison failed
  Obtained: 967121084.0761821
  Expected: 1018975372 � 1.0e+07
Not sure why the "Checks / Test" part of CI is passing with the Windows solver finding the same optimal point.

@dguittet the main difference between "Tests (non-editable install)" and "Tests" is that the former might not have access to some of the data (non-.py) files if they haven't been registered in setup.py.

Is it at all possible that this might be related with the failure? I see that there are many data files being added with this PR, but only a subset of them is listed in setup.py. I understand that this might not matter if the presence of the files makes no difference at all in terms of the solver, but thought it makes sense to at least mention this as a possible cause of the discrepancy.

dguittet commented 1 year ago

@lbianchi-lbl I don't think that's the reason since the Linux non-editable install tests are passing

lbianchi-lbl commented 1 year ago

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
idaes-pse 2.0.0 requires pandas<1.5, but you have pandas 2.0.3 which is incompatible.

I got the following error when I ran RE_surrogate_optimization_steadystate.py

(test-fe-models) C:\Users\Radhakrishna\Desktop\code_repositories\dispatches\dispatches\case_studies\renewables_case>python RE_surrogate_optimization_steadystate.py
Traceback (most recent call last):
  File "C:\Users\Radhakrishna\Desktop\code_repositories\dispatches\dispatches\case_studies\renewables_case\RE_surrogate_optimization_steadystate.py", line 340, in <module>
    result = run_design()
  File "C:\Users\Radhakrishna\Desktop\code_repositories\dispatches\dispatches\case_studies\renewables_case\RE_surrogate_optimization_steadystate.py", line 293, in run_design
    model, n_rep_days = conceptual_design_dynamic_RE(default_input_params, PEM_bid=PEM_bid, PEM_MW=PEM_size, verbose=False)
  File "C:\Users\Radhakrishna\Desktop\code_repositories\dispatches\dispatches\case_studies\renewables_case\RE_surrogate_optimization_steadystate.py", line 101, in conceptual_design_dynamic_RE
    net_rev_defn, net_frequency_defn, dispatch_clusters_mean, pem_clusters_mean, resource_clusters_mean = load_surrogate_model(re_nn_dir)
  File "C:\Users\Radhakrishna\Desktop\code_repositories\dispatches\dispatches\case_studies\renewables_case\RE_surrogate_optimization_steadystate.py", line 59, in load_surrogate_model
    model = pickle.load(f)
ModuleNotFoundError: No module named 'sklearn'

Hi @radhakrishnatg, those errors are unexpected if the scripts were run in a fresh environment after installing using pip install -r requirements-dev.txt. Can you provide more details on the environment where those errors occurred?

I agree that one of the "implicit" dependencies required to read parquet files should be added if any of the scripts uses that format.

radhakrishnatg commented 1 year ago

Hi @lbianchi-lbl, Sorry, I lost track of your comment. I did not install install DISPATCHES using pip install -r requirements-dev.txt. I installed it using pip install dispatches to test if the code is working with the release candidate or not. By details of the environment, I'm assuming that you mean the list of installed packages. Please let me know if that is not the case:

(test-fe-models) C:\Users\Radhakrishna>pip freeze
absl-py==1.4.0
anyio==3.7.1
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
astunparse==1.6.3
attrs==23.1.0
backcall==0.2.0
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.12.2
bleach==6.0.0
blinker==1.6.2
bunch==1.0.1
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.4
colorama==0.4.6
comm==0.1.3
contourpy==1.1.0
coramin==0.1.1
cramjam==2.6.2
cycler==0.11.0
debugpy==1.6.7
decorator==5.1.1
defusedxml==0.7.1
dispatches==1.3.0rc0
dispatches-data-packages==23.3.19
dispatches-rts-gmlc-data==23.6.30
distro==1.8.0
et-xmlfile==1.1.0
exceptiongroup==1.1.2
fastjsonschema==2.17.1
fastparquet==2023.7.0
Flask==2.3.2
Flask-Cors==4.0.0
flatbuffers==23.5.26
fonttools==4.40.0
fqdn==1.5.1
fsspec==2023.6.0
gast==0.4.0
google-auth==2.21.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
graphviz==0.20.1
gridx-egret==0.5.5
gridx-prescient==2.2.2
grpcio==1.56.0
h5py==3.9.0
idaes-pse==2.0.0
idna==3.4
importlib-metadata==6.7.0
importlib-resources==5.12.0
iniconfig==2.0.0
ipykernel==6.24.0
ipython==7.34.0
ipython-genutils==0.2.0
ipywidgets==8.0.7
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
jsonpointer==2.4
jsonschema==4.18.0
jsonschema-specifications==2023.6.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.6.3
jupyter_client==8.3.0
jupyter_core==5.3.1
jupyter_server==2.7.0
jupyter_server_terminals==0.4.4
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.8
keras==2.13.1
kiwisolver==1.4.4
libclang==16.0.0
lxml==4.9.3
Markdown==3.4.3
MarkupSafe==2.1.3
matplotlib==3.7.2
matplotlib-inline==0.1.6
mistune==3.0.1
mpmath==1.3.0
nbclassic==1.0.0
nbclient==0.8.0
nbconvert==7.6.0
nbformat==5.9.0
nest-asyncio==1.5.6
networkx==3.1
notebook==6.5.4
notebook_shim==0.2.3
NREL-PySAM==4.2.0
numpy==1.24.3
oauthlib==3.2.2
omlt==1.1
openpyxl==3.1.2
opt-einsum==3.3.0
overrides==7.3.1
packaging==23.1
pandas==2.0.3
pandocfilters==1.5.0
parso==0.8.3
pickleshare==0.7.5
Pillow==10.0.0
Pint==0.22
platformdirs==3.8.1
pluggy==1.2.0
ply==3.11
prometheus-client==0.17.0
prompt-toolkit==3.0.39
protobuf==4.23.4
psutil==5.9.5
pyarrow==12.0.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
Pygments==2.15.1
Pyomo==6.5.0
pyparsing==3.0.9
pytest==7.4.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-json-logger==2.0.7
python-slugify==8.0.1
pytz==2023.3
pywin32==306
pywinpty==2.0.10
PyYAML==6.0
pyzmq==25.1.0
qtconsole==5.4.3
QtPy==2.3.1
rbfopt==4.2.6
referencing==0.29.1
requests==2.31.0
requests-oauthlib==1.3.1
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.8.8
rsa==4.9
scipy==1.11.1
seaborn==0.12.2
Send2Trash==1.8.2
six==1.16.0
sklearn==0.0.post7
sniffio==1.3.0
soupsieve==2.4.1
sympy==1.12
tensorboard==2.13.0
tensorboard-data-server==0.7.1
tensorflow==2.13.0
tensorflow-estimator==2.13.0
tensorflow-intel==2.13.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.3.0
terminado==0.17.1
text-unidecode==1.3
tinycss2==1.2.1
tinydb==4.8.0
tomli==2.0.1
tornado==6.3.2
traitlets==5.9.0
typing_extensions==4.5.0
tzdata==2023.3
uri-template==1.3.0
urllib3==1.26.16
utm==0.7.0
wcwidth==0.2.6
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.1
Werkzeug==2.3.6
widgetsnbextension==4.0.8
wrapt==1.15.0
xlrd==2.0.1
zipp==3.15.0

lbianchi-lbl commented 1 year ago

Hi @lbianchi-lbl, Sorry, I lost track of your comment. I did not install install DISPATCHES using pip install -r requirements-dev.txt. I installed it using pip install dispatches to test if the code is working with the release candidate or not.

OK, thanks for clarifying. I'm still a little confused as I see that dispatches==1.3.0rc0 is in the environment. However, until the final 1.3.0 release is cut, I'd expect pip install dispatches to result in the latest stable release to be installed, i.e. 1.2.0. Is there a chance that the command used for this environment was different, e.g. pip install dispatches --pre or pip install dispatches==1.3.0rc0? (this is not an essential detail; I'm just trying to see if maybe there's some issue with the release candidate).

In any case, pip install dispatches in a fresh environment would not install the optional dependencies that are needed for the surrogate models. This would be consistent with the sklearn: module not found error, since scikit-learn is one of those optional dependencies. Optional dependencies are installed with pip install -r requirements-dev.txt.

dguittet commented 1 year ago

@lbianchi-lbl I added fastparquet to the requirements-dev.txt

dguittet commented 1 year ago

@radhakrishnatg @Xinhe-Chen @lbianchi-lbl Ready for review

dguittet commented 1 year ago

(2) Do you need all the Prescient outputs in order to run your code? These data files can be massive, so if possible we would like to avoid including them in the man repo. We can push the csv files to the data repository just like what we did with the RTS-GMLC dataset. @lbianchi-lbl any thoughts on this?

@radhakrishnatg I've consolidated the Prescient outputs so there should be fewer data files remaining now

dguittet commented 1 year ago

Thanks for the review @radhakrishnatg !

dguittet commented 1 year ago

@adowling2 Can you please re-review?

ksbeattie commented 11 months ago

@dguittet & @adowling2: any reason not to merge this? We need to cut this final final release to completely wrap this up.

dguittet commented 11 months ago

@ksbeattie I don't know of any reason not to merge this

adowling2 commented 11 months ago

I'm okay with merging.

gmlc-dispatches / dispatches

Re paper #208

Addresses issue:

Legal Acknowledgement

Codecov Report