Change KAPy config to suit our needs

[ketilt]

Challenge

• Current KAPy setup assumes a set of five distinct collections of datasets, see excerpt from config.yaml below.
• These categories are referenced and used in the core KAPy functions under workflows/KAPy
• In our KiN flow chart, however, we have 13 distinct collections of datasets (excluding published data):
  • Unchecked reference data
  • Unchecked downscaled data
  • Unchecked hydrological variables
  • Unchecked climate indices
  • Unchecked 30-year statistics
  • Unchecked ensemble statistics
  • Reference data
  • Climate variables
  • Climate indices
  • 30-year statistics
  • Ensemble statistics
  • Timeseries files
  • Image files
• Each processing step we implement, has one of these 13 as input and one of them as output.
• Other pipelines are imaginable with more or fewer collections

Our needs?

• Test cases should generally source input files from KiN folders, and store output files in Klimakverna folders.
• Users should be able to override input and output paths when triggering a pipeline (test cases 6 and 8).

Suggestion

• Expand config.yaml to fit the 13 collections in KiN
• Change KAPy core functions if necessary to always save in QA quarantine (gray location in flow chart)
  • Except QA itself, which can save to appropriate output folders (green location in flow chart)

Details

Excerpt from config/config.yaml:

configurationTables:
    inputs:  'config/inputs.tsv'
    indicators: 'config/indicators.tsv'
    scenarios:  'config/scenarios.tsv'
    periods:  'config/periods.tsv'
    seasons: 'config/seasons.tsv'
dirs:
    variables: 'results/1.variables'
    indicators: 'results/2.indicators'
    regridded: 'results/3.commmon_grid'
    ensstats: 'results/4.ensstats'
    arealstats: 'results/5.areal_statistics'
    plots: 'results/6.plots'