Test routine: add Makefile; CI: consolidate ci.yaml files

[FabianHofmann]

Closes #1052

Changes proposed in this Pull Request

This PR proposes four changes:

• fix a bug in add_extra_components. CSP generators are not necessarily attached to every bus (this is probably related to a "missing" substation or shape assigment). This lead to an error in the CI before (as mentioned in #1052) in the alignment of buses_i being the AC-buses and the csp buses.
• use a Makefile to setup and run tests. With make test all the snakemake tests can now be run locally. Why is that better? The workflow used a recursive snakemake call. This lead to errors not being reported back as errors. With the Makefile, errors are strightly reported back and the test command are explicit and easily accessible.
• the run_scenario rule now streams errors which it did not do before. At some point, we might also think using something similar as for the tests to improve visibility of the command used.
• Consolidate CI yaml. Use only one CI yaml instead of three. Use micromamba with it built in cashing functionality.

In general, the workflow would profit from using --configfile=<additional.config.yaml> instead of the diff_config approach. Was there a specific reason to stick to the manual insertion of config diffs? Perhaps is that historically grown? Note that --configfile overrides existing and adds additonal keys to the config given in the Snakefile. So it does essentially the same.

The CI will likely fail now, I need probably some more iterations.

Checklist

• [x] I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• [x] I tested my contribution locally and it seems to work fine.
• [x] Code and workflow changes are sufficiently documented.
• [x] Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• [x] Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• [x] Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• [x] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• [x] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[pz-max]

I can comment here a bit:

• @FabianHofmann , we split the CI to test everything in Linux (faster) & only a subset in other OS. Another reason was the need for a badge for each OS. This can be useful e.g. when you run on Linux where all tests are passing and you don't care about Windows too much.
• love the make solution

Regarding this bit:

In general, the workflow would profit from using --configfile= instead of the diff_config approach. Was there a specific reason to stick to the manual insertion of config diffs? Perhaps is that historically grown? Note that --configfile overrides existing and adds additonal keys to the config given in the Snakefile. So it does essentially the same

I worked on the config diff stuff. Snakemake uses the exact same config merging code as we do (realised that later). I think @davide-f possibly added a few other parts. We implemented it like this to be independent of Snakemake and more flexible for this functionality (e.g. we are using ruaml (as Yaml container), or smt like that, which also keeps comments after the config is merged). Not sure what other benefits/ disadvantages exist. @davide-f might have more thoughts.

Feel free to decide what is the better solution for now :+1:

[FabianHofmann]

@pz-max @davide-f this is ready for being merged, objective values are reproduced. A merge today would be cool, so I can look into #796 soon

[finozzifa]

Indeed a great PR @FabianHofmann! thanks a lot

[FabianHofmann]

@davide-f @ekatef the PR is now ready. As discussed the config.default.yaml is ignored by now, and if no config.yaml is existent, the config.tutorial.yaml is copy to config.yaml

[ekatef]

@davide-f @ekatef the PR is now ready. As discussed the config.default.yaml is ignored by now, and if no config.yaml is existent, the config.tutorial.yaml is copy to config.yaml

Hey @FabianHofmann, thank you so much for the discussion and the adjustments! I love the solution with commenting-out the automated update using config.default.yaml. Absolutely agree that improvements are needed on management of the configuration files, and having this like in Snakemake should be an effective reminder! Could we probably add an issue to track the ideas and the progress on that, also accounting for the modularity idea @davide-f has suggested?

[FabianHofmann]

@ekatef that is a good idea! thank you for your comment. I will raise an issue

[davide-f]

Thanks @FabianHofmann ! Agree, let's keep it like this for now and edit after the merge :)

[FabianHofmann]

mmh, annoying, as soon as passing the --configfile via snakemake, the condition

if "config" not in globals() or not config:  # skip when used as sub-workflow

is false, so it will not load the config.yaml. It reverted the latest changes and have a look on monday, whether the if clause is really necessary for pypsa-earth-sec, or how we can adjust it.

[davide-f]

mmh, annoying, as soon as passing the --configfile via snakemake, the condition

if "config" not in globals() or not config:  # skip when used as sub-workflow

is false, so it will not load the config.yaml. It reverted the latest changes and have a look on monday, whether the if clause is really necessary for pypsa-earth-sec, or how we can adjust it.

Unfortunately it is necessary, I had to add that if clause exactly because of that unfortunately. Worth pointing @hazemakhalek to agree on a solution. If we merge this, we cannot update -sec to latest main here The merge will solve this issue...

[davide-f]

Hello @FabianHofmann , just to confirm, we can close this PR as it is merged already into the merge right?

[FabianHofmann]

Correct!