Memory handling becomes an issue for large model instances

[jokochems]

For solving large long-term models (European scope, 2020-2025/2050, hourly resolution, including demand response), currently Out of Memory Errors are faced.

These are twofold:

Problem 1 (full instance: including demand response and 2020-2050):

• An Out of Memory Error already occurs in the model build. Thus, the model formulation itself causes some problems.
• Potential fixes:
  • (Re-)formulate the model to be more memory-efficient. → Requires some fundamental adjustments and perhaps touching the underlying pyomo / oemof.solph architectures and thus is not preferred.
  • Reduce the model. → Two kinds of reductions seem feasible for my use case of pommesinvest: Reduce the optimization horizon to 2020-2045 (as has already been done) and/or aggregate market zones of other countries to one single geographical node per country as the focus is on evaluating Germany and cross-border exchange is not of major interest.

Problem 2 (reduced instance: reducing to 2020-2045):

• When solving the model (presolve), an out of memory error occurs since the temporary LP-File to be stored in /tmp is larger than the allocated space there.
• Potential fixes:
  • Utilize pyomos TempfileManager to address this issue.
  • Re-locate the directory to store temp files to disk (which slows down computation, but brings more memory space).
  • Both options to fix problem 2 seem feasible and might also be used in combination.

[jokochems]

13 might solve problem 2, while for problem 1, as written, this might be resolved by adjusting the model configuration itself. Thus, at first, no further code changes seem necessary. This issue should nevertheless be reopened if needed.

[jokochems]

Another heads up: Using this strategy, a model run with hourly resolution and considering the median estimate for technical demand response potentials as well as for the costs of flexibility options terminated, though memory usage (around 460 GB) was very close to the available resource limit.