search

calliope-project / euro-calliope

A workflow to build models of the European electricity system for Calliope.
https://euro-calliope.readthedocs.io
MIT License
31 stars 17 forks source link

Make `rule all:` lean. #305

Open irm-codebase opened 2 months ago

irm-codebase commented 2 months ago

What can be improved?

The problem

Currently, the main Snakefile produces files for different model configurations (national, regional, continental).

This results in a lot of wasted processing time, and usually leads to files you do not really need.

rule all:
    message: "Generate euro-calliope pre-built models and run tests."
    input:
        "build/logs/continental/test.success",
        "build/logs/national/test.success",
        "build/models/continental/example-model.yaml",
        "build/models/national/example-model.yaml",
        "build/models/regional/example-model.yaml",
        "build/models/build-metadata.yaml",
        "build/models/regional/summary-of-potentials.nc",
        "build/models/regional/summary-of-potentials.csv",
        "build/models/national/summary-of-potentials.nc",
        "build/models/national/summary-of-potentials.csv",
        "build/models/continental/summary-of-potentials.nc",
        "build/models/continental/summary-of-potentials.csv"

rule all_tests:
    message: "Generate euro-calliope pre-built models and run all tests."
    input:
        "build/models/continental/example-model.yaml",
        "build/models/national/example-model.yaml",
        "build/models/regional/example-model.yaml",
        "build/logs/continental/test.success",
        "build/logs/national/test.success",
        "build/logs/regional/test.success",
        "build/models/build-metadata.yaml",
        "build/models/regional/summary-of-potentials.nc",
        "build/models/regional/summary-of-potentials.csv",
        "build/models/national/summary-of-potentials.nc",
        "build/models/national/summary-of-potentials.csv",
        "build/models/continental/summary-of-potentials.nc",
        "build/models/continental/summary-of-potentials.csv"

The solution

Files generated should depend on the used configuration (which should be either continental or minimal by default). Here, config["setup"]["resolution"] = "continental".

rule all:
    message: "Generate euro-calliope pre-built models and run tests."
    input:
        f"build/logs/config["setup"]["resolution"]/test.success",
        "build/models/config["setup"]["resolution"]/example-model.yaml",
        "build/models/build-metadata.yaml",
        "build/models/config["setup"]["resolution"]/summary-of-potentials.nc",
        "build/models/config["setup"]["resolution"]/summary-of-potentials.csv"

Version

1.0.0

timtroendle commented 2 months ago

Maybe we can build it in a way that the config param is a list that can hold 1--3 resolutions? As a user, I often use more than one resolution: one for rapid prototyping (typically continental) and one (or two) for the actual analysis.

irm-codebase commented 2 months ago

Nice idea. Snakemake's expand function should do the trick https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function