dib-lab / farm-notes

notes on the farm cluster
16 stars 9 forks source link

lessons from charcoal - running big snakemake on farm #34

Open ctb opened 2 years ago

ctb commented 2 years ago

@taylorreiter quoth:

lower run times on bml make jobs run faster. I know this is sort of an obvious statement, but reducing from 8 hours to 90 minutes was orders of magnitude speed up in how soon my jobs started actually running

-j 200 is my friend. farm can handle it and it makes things go faster

splitting charcoal conf files to handle 1/20th of gtdb at a time also had orders of magnitude speed up for dag solve time and makes no difference to me on analyzing the output results