jhrcook
commented
2 years ago I'm only able to control cluster job-submission order by adding resources to the jobs and then assigning different amounts of some arbitrary resource given the job's wildcards. Running Snakemake with a CLI parameter for the total amount of available resources then informs the order of job submission. This is still imperfect for a few reasons, namely that Snakemake waits to submit jobs until it thinks the resources are available. I still want the jobs all submitted, just in a specific order.
Because of the success in using the GPUs for sampling on O2 (#177), I want to refactor the model fitting pipeline to use the GPUs.
I need to be aware of the more restrictive resource allocations – I will not be able to run all models simultaneously. It may be better to run chains of the same final model in order so that they will tend to finish around the same time. I need to see what options Snakemake has for implementing this.
Using O2 GPU resources How to choose a partition in O2