search

PyPSA / pypsa-usa

PyPSA-USA: An Open-Source Energy System Optimization Model for the United States
https://pypsa-usa.readthedocs.io
MIT License
35 stars 15 forks source link

Move retrieval of Historical EIA balancing data out of `retrieve_eia` rule #253

Closed trevorb1 closed 2 months ago

trevorb1 commented 2 months ago

Checklist

The Issue

The workflow keeps getting stuck at retrieval of Grid Emissions data. It was working fine for me yesterday evening, so not sure what changed. Il try again tomorrow to see if it was maybe just a connection thing.

This error has lasted for a few hours now though, and happens on the master branch. I can still access other API calls okay (such as connecting to the EIA API).

Steps To Reproduce

Not sure how reproducible this will be, but I am just using the default config file to run the workflow.

Expected Behavior

No response

Error Message

Error in rule retrieve_eia_data:
    jobid: 22
    output: data/GridEmissions/EIA_DMD_2018_2024.csv
    log: logs/retrieve/retrieve_historical_load_data.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /home/trevor/repos/pypsa-usa/workflow/rules/retrieve.smk, line 133:
Command 'set -euo pipefail;  /home/trevor/miniconda3/envs/pypsa-usa/bin/python3.11 /home/trevor/repos/pypsa-usa/workflow/.snakemake/scripts/tmpiajirjed.retrieve_eia_data.py' returned non-zero exit status 1.
  File "/home/trevor/repos/pypsa-usa/workflow/rules/retrieve.smk", line 133, in __rule_retrieve_eia_data
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/concurrent/futures/thread.py", line 58, in run

Traceback (most recent call last):
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 1390, in getresponse
    response.begin()
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 1390, in getresponse
    response.begin()
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/http/client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/repos/pypsa-usa/workflow/.snakemake/scripts/tmpiajirjed.retrieve_eia_data.py", line 223, in <module>
    download_csvs(urls, PATH_DOWNLOAD_CSV)
  File "/home/trevor/repos/pypsa-usa/workflow/.snakemake/scripts/tmpiajirjed.retrieve_eia_data.py", line 49, in download_csvs
    response = requests.get(url)
               ^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniconda3/envs/pypsa-usa/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Anything else?

No response

trevorb1 commented 2 months ago

The issue was not with the GridEmission data, but the retrieval of historical EIA data, which is used for validation. This is commented out in PR #251