search

PyPSA / pypsa-usa

PyPSA-USA: An Open-Source Energy System Optimization Model for the United States
https://pypsa-usa.readthedocs.io
MIT License
35 stars 15 forks source link

Change processing of EFS data to xarray #286

Open trevorb1 opened 2 months ago

trevorb1 commented 2 months ago

Feature Request

With the introduction of ReadEfs in PR #202, more EFS data is retained for sector coupling studies. The issue is, if many sectors are using the EFS data and try to process it at the same time, (I think) memory consumption skyrockets as I am sometimes (and seemingly randomly) getting killed errors when running. This has not been an issue for electricity only studies or if running the script by itself.

Suggested Solution

I believe xarray will be much more memory efficient at processing EFS data.

Additional Info

No response

ktehranchi commented 2 months ago

One way we could significantly reduce data size significantly would be to aggregate to substations earlier on in the workflow. Right now bus:sub is a many:1 ratio where all buses have the same GPS location based on the the substation. This would involve making many modifications throughout the workflow, so no quick task. We would probably want to change build_base_network to aggregate bus to sub.