New IO standard

[irm-codebase]

Fixes #30 and #14

IO concept

IO standard to make modules easier to use on the user/model side, which meets snakemake's recommended structure.

The preexisting hydropower module and the module template were updated to showcase it.

Inputs

• module/resources is where all 'raw' data needed for the module to work is deposited. This is downloaded data or user inputted data.
  • module/resources/user_input is were all user inputted files should land.
    • In most cases the input will be shapefiles with unique names (ehighways.geojson, DEU.geojson, MEX.geojson...).
    • Wildcards are used to 'trigger' the rest of the processing.

Configuration

• use_default_user_input is a module-level configuration bool that, when True, will:
  • activate rules that download default files into resources/user_input
  • activate rule all, which triggers the module's processing for the default input.
• otherwise, these rules are completely removed from the workflow, meaning the module will only be 'triggered' if a user explicitly inputs a file and requests results in their rules.

Outputs

Check the hydropower README for detailed info. A brief summary:

• module/results is were all the 'processed' files land.
• Results are arranged using wildcards to avoid conflicts as much as possible, and to re-use results that are not wildcard dependent.

[irm-codebase]

Resulting structure when running in euro-calliope-modular

units is the name of the resolution used by euro calliope, so it's the name used for the folders. If instead someone passed files for national.geojson, ehighways.geojson, etc; they'd have separate folders for each.

Please note that 2016 is just a wildcard. This makes it possible to request data for different resolutions for different years.

module_hydropower
├── resources
│   ├── basins
│   │   └── raw_eu.geojson
│   ├── Geth_2015_PHS_capacity.csv
│   ├── IRENA_hydro_generation.csv
│   ├── jrc_hydropower_plants.csv
│   └── user_input
│       └── units.geojson
└── results
    ├── basins
    │   └── preprocessed_eu.gpkg
    ├── jrc_hydropower_plant_database_preprocessed.csv
    ├── plots
    │   ├── basins
    │   │   └── basin_eu.png
    │   └── units
    │       └── 2016
    │           └── plot_cutout.png
    └── shapes
        └── units
            ├── 2016
            │   ├── capacity_factors_reservoir.csv
            │   ├── capacity_factors_ror.csv
            │   ├── cutout.nc
            │   ├── hydropower_inflow_energy.nc
            │   └── hydropower_inflow_m3.nc
            ├── hydropower_storage_capacity.csv
            └── hydropower_supply_capacity.csv

13 directories, 17 files

[irm-codebase]

@jnnr Thanks for the feedback!

Couple of clarifications, based on our discussion:

• In resources/, by user inputted data I meant resources/user_input. I updated the original comment to make this less confusing.
• User inputs is anything external that the module needs to work. Some of it might be customisable (shapes, regional capacity, population per region...), some might not (private datasets like IEA energy balances).

I think renaming user_input makes sense for the points you raised. However:

• customizable_resources repeats the naming of the top level (resources/customizable_resources), and will lead to long paths in the code...

How about resources/customisable/ | resources/custom? And if we have non-customisable resources that users must give, we just tell them to drop it in resources/ with a specific name.

[irm-codebase]

@jnnr I've updated the module to the agreed convention. Please let me know if I can merge this so the templates are updated.