cnluzon
commented
4 years ago I have not compared these, it did not occur to me to run the pipeline on a single node. We have done local runs on my desktop computer in the lab, which is not as powerful as an Uppmax node but not extremely far from it, and the timing is definitely longer but it may be an option for jobs that are not very large.
I think implementing #73 can balance these two options. There are many jobs that could be run on the snakemake running job, making queues less busy and potentially putting less stress on the file system, since jobs on the snakemake job would be run sequentially.
marcelm
(The job is marked as failed because of an error in the final 
@cnluzon I noticed that you are running the pipeline in cluster mode (submitting individual jobs to the cluster, running snakemake only on the head node). I’d like to mention that there’s an alternative, which is to run the entire pipeline as one batch job on a single node. Have you compared these two methods?
Of course, the advantage of submitting jobs to the cluster is that it is potentially faster, but there are some disadvantages:
/scratchdirectory of a single node and copy everything to the target folder after we’re done, and then we wouldn’t need to add special code to each rule to move stuff into and out of$TMPDIR(see #71).Also, when submitting individual jobs to the cluster, each individual job is going to sit in the queue, which increases (wall-clock) runtime, so it isn’t so obvious that submitting to the cluster is actually that much faster.
Perhaps running one big job is out of the question because it won’t finish in reasonable time on a single node, but we should be aware that it’s a tradeoff.