Add csv-demand parser

[ekatef]

987

The aim of the PR is to provide an option to use csv files as demand inputs.

Checklist

• [x] I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• [x] I tested my contribution locally and it seems to work fine.
• [ ] Code and workflow changes are sufficiently documented.
• [ ] Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• [ ] Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• [ ] Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• [ ] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• [x] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

CI currently fails during building a test configuration, when get_load_paths_gegis is called from Snakemake. It looks like there may be some difference in naming conventions.

[ekatef]

The tutorial workflow in CI fails due to an empty power plants matching located in the requested region. Not sure in which way it is connected with the changes introduced by the PR, and can't reproduce it locally yet.

[ekatef]

@davide-f could you please have a look? CI is failing due an some troubles in finding power plants for the region, which I can't reproduce locally... Any ideas be very welcome on what may go wrong!

[ekatef]

The changes in this PR should be aligned with #1017.

[ekatef]

CI fails on building the test configuration in the tutorial run complaining that it's not able to find the demand input looking into 'data/ssp2-2.6/2030/era5_2013/Africa.nc'. The reason is that there is still no data files on the stage of the configuration building. Replaced an error with a warning.

[ekatef]

The issue has been linked with the fact that the load paths list has been formed by CI before loading any data. As a result, all the existence checks were negative, and a list of load files empty.

Have modified definition of the load paths in the Snakemake, adding tracking of load parameters of Snakemake.params.

[ekatef]

Tested successfully on both nc-only inputs and a combination of nc and csv.

The current implementation changes the architecture of Snakemake workflow, as the previous one was designed to run also if the demand inputs don't exist yet. Not sure the current implementation is the best possible, and happy to revise.

@davide-f would be very grateful for your review 🙂

[ekatef]

Great @ekatef :D The functionality is absolutely there!

Need to check why the CI is not working though

As minor comments:

• plase revise the release_notes
• it may be good to add something in the documentation to clarify on how to feed in the custom data and leverage on this amazing contribution. I understand this is time demanding. not sure if you have the time to craft something very very basic to be later improved or we leave it completely for a next PR.

What do you think?

Thanks a lot for reviewing @davide-f! 😄 I can confirm that the functionality has been tested successfully :) 🎉

The reason for CI failing has been that Snakemake is parsing all the inputs before starting to do anything. So, it didn't help much to place get_load_paths_gegis() in the inputs of build_demand_profiles, as Snakemake still executed get_load_paths_gegis() before loading the databundle when running the tutorial case. That leads to failure along the workflow, if a new modification of get_load_paths_gegis is used as it captures only the paths which really exist of the drive.

To fix this, I had to put back composing of load_data_paths, but add a condition to treat the paths differently, depending on the actual presence of the load data. Not sure it's the most elegant approach but seems to be an effective one.

Done for a release note :D Have also drafted a documentation on using custom load profiles. Could you please check if this addition looks clear enough to be used?

PS Can't help but say that it's amazing to have a documentation build as a part of CI 💚 Thanks a lot to you and @GbotemiB for adding this feature!

[ekatef]

Cool, the CI is running :D so closer to merge :)

@davide-f thanks :D Have implemented your suggestions, and the changes look now quite consistent with the original version which feels really great 🙃 Thanks a lot for reviewing!

Both local and CI testing are successful now.

Could you please check if everything looks well in the revised version?