Snakemake modifications for big models

KTH-dESA / esom_gsa

snakemake workflow for performing a global sensitivity analysis of an OSeMOSYS model

MIT License

7 stars 3 forks source link

Snakemake modifications for big models #10

Open Timon-R opened 7 months ago

Timon-R commented 7 months ago

I used the workflow on a big model with 300 runs and had to make adjustments to the snakemake workflow. I note them here for convenience.

Change shell commands to script commands when there are multiple arguments. Command line commands have limited arguments and if lists are given, this can cause the workflow to break.
Use priority: 100 for the function that solves the lp file as well as for all functions that process it
and use priority 0 for the function that creates it; also give the function that creates the lp file 4 threats. Reason for this is that the lp file is large and can take up a lot of space on your hard drive.

robertodawid commented 1 month ago

Hi @Timon-R , I am running the snakemake workflow in Windows. It seems there are issues with the (shell sections in the workflow) since most of these sections were developed for *nix. I am NOT super literate on this, but you mentioned changing the shell commands to scrips. Would that work on Windows? If so, do you have an example I can look at? or in any case. Would you happen to have a suggestion so I can run the workflow in Windows.?

Timon-R commented 1 month ago

Hi @robertodawid, try using my modified workflow for Windows: https://github.com/Timon-R/OSEMBN

robertodawid commented 1 month ago

Hi @Timon-R , I am running the snakemake workflow in Windows. It seems there are issues with the (shell sections in the workflow) since most of these sections were developed for *nix. I am NOT super literate on this, but you mentioned changing the shell commands to scrips. Would that work on Windows? If so, do you have an example I can look at? or in any case. Would you happen to have a suggestion so I can run the workflow in Windows.?

Dear @Timon-R , Thanks! I took a couple of the *.py from your workflow, and now mine seems to be working.