Move from per-storm to per-sample jobs - Githubissues

nismod / open-gira

Open-data Global Infrastructure Risk/Resilience Analysis

https://nismod.github.io/open-gira/

MIT License

12 stars 3 forks source link

Move from per-storm to per-sample jobs #169

Closed thomas-fred closed 6 months ago

thomas-fred commented 6 months ago

Reduce snakemake's burden by lowering the number of jobs it has to create and track:

Move from per-storm jobs to per-sample ones. Running a global study is now hundreds of jobs, not ~1 million.
Synthetic track sets (STORM, IRIS) are split into ten 1,000 year 'samples', 0 to 9.
IBTrACS is given a single sample number, 0, containing all historic tracks.

Also:

Remove usage of output_dir key in config. Instead, infer output directory from file paths' OUTPUT_DIR wildcard.
Function and rule to merge tables of different GADM levels to provide e.g., level 2 where available, otherwise level 1 otherwise level 0.