search

PyPSA / pypsa-ariadne

MIT License
6 stars 4 forks source link

Set up config for CI runs #111

Open lindnemi opened 2 weeks ago

lindnemi commented 2 weeks ago

Just 1 scenario, no solar thermal.

Still this workflow takes ~30 minutes locally due to all the pre-processing. I guess some further simplifications are necessary for the CI runs, e.g. disabling "estimate_renewables" and using a default cutout. Here i would need some input from @fneum

lkstrp commented 2 weeks ago

@lindnemi config/scenarios.automated.yaml is set in the config bus but is missing in the repo

lindnemi commented 2 weeks ago

we usually generate it via

snakemake -j1 build_scenarios
lindnemi commented 1 week ago

I talked to Fabian and there are basically two ways to reduce the runtime of the workflow:

  1. Use Github CI with Caching: The most compute intensive rule at the moment is build_renewable_profiles. We could cache the output similiar to how it is done here in the pypsa-eur repo:

https://github.com/PyPSA/pypsa-eur/blob/450f75a989dd9b84e27a7690a96107bdca4e8035/.github/workflows/ci.yaml#L75C1-L81C71

  1. Forward the CI job to the z1 cluster and execute it there. @lkstrp would you know how to do that? The upside would be that we can use more resources and compute multiple scenarios.
lkstrp commented 1 week ago

Option 1 does not work because of disk space issues in a GitHub runner. There are some hacks around this, and using a default cutout would also help, but I think it's better to set up the workflow straight with the cluster. Also to be more future proof. I will dig into it.