SLURM: Option to join stdout and stderr

evoldoers / biomake

GNU-Make-like utility for managing builds and complex workflows

BSD 3-Clause "New" or "Revised" License

103 stars 9 forks source link

SLURM: Option to join stdout and stderr #58

Open sjackman opened 6 years ago

sjackman commented 6 years ago

If SLURM's bsub -e option (standard error file) is not specified, the default is to join stdout and stderr, which is my preferred behaviour. See http://www.glue.umd.edu/lsf-docs/man/bsub.html

sjackman commented 6 years ago

Is there a way to achieve this behaviour by modifying QsubArgs, without adding an option to biomake?

sjackman commented 6 years ago

You have saved my :bacon:. I have run 2,344 jobs in the last week with biomake -Q slurm, and I never would have been able to keep them all straight or get the analysis done in time if I had been scheduling them manually. ❤️ 👏 🤗

sjackman commented 6 years ago

Could biomake -Q poolq optionally redirect joined stdout/stderr to .biomake/poolq/out/$@? I'd also find it quite useful to write the PID of active processes to .biomake/poolq/job/$@. I've fallen in love with using pstree and pv -pd $! to inspect and monitor active processes. If you haven't encountered pv -pd $! before, check it out! It's amazing. http://www.ivarch.com/programs/pv.shtml