search

legend-exp / legend-dataflow

LEGEND data flow management
Other
2 stars 9 forks source link

check simultanous snakemake executions #5

Closed mmatteo closed 2 months ago

mmatteo commented 1 year ago

Check whether multiple instances of snakemake can run at the same time sharing the .snakemake directory. This would be useful to process different data sets at the same time

gipert commented 9 months ago

In principle yes, I am doing this to process simulations separately in legend-simflow. But you need to make sure the sub-workflows are completely independent, otherwise Snakemake will detect this and fail.

gipert commented 2 months ago

https://snakemake.readthedocs.io/en/stable/executing/cli.html#dealing-with-very-large-workflows would be also useful to split the workflow into batches to be sent to different computing nodes (at e.g. NERSC)