Configuration for virsorter 2 with LSF ?

[michalogit]

Hi Jiarong et al,

It was good to see that internally there is a snakemake workflow, so I have tried it already on our HPC cluster with LSF queueing – works happily on a small test dataset.

Do you have by chance a snakemake configuration in a form of a yaml file for the LSF profile (or for slurm, SGE…), specifying how eg many cores and how much memory specific snakemake rules would need to run in a balanced way? Or any hints on choosing cores/memory/running time for particular rules would be helpful. I think your default configuration was tuned to a one large server, right?

Cheers, Michal

[jiarong]

Hi Michal, I have not optimized the workflow for clusters and worked on the cluster profile yet (virsorter run --profile). I recommend running in single node and give it enough cores for now. If you want to try cluster mode, all the rules should use memory < 10Gb, and walltime < 2 hours except for hmmsearch (8 hours).

[jiarong]

I forgot to say about cores. All rules are single core, except hmmsearch. You can change hmmsearch core # with HMMSEARCH_THREADS in template-config.yaml, which should give linear speed up on hmmsearch up to 4 cores and then efficiency start to drop.

[michalogit]

Thank you Jiarong!

I had some doubts about cores settings, as my cluster logs report "max threads" like this:

LSBATCH: User input /cluster/scratch/michalo/test_vs/test4/.snakemake/tmp.0d587yl9/snakejob.hmmsearch.218.sh

Successfully completed. Resource usage summary: CPU time : 489.86 sec. Max Memory : 526 MB Average Memory : 301.83 MB Total Requested Memory : 1024.00 MB Delta Memory : 498.00 MB Max Swap : - Max Processes : 6 Max Threads : 9 Run time : 495 sec. Turnaround time : 502 sec.

In "gff_feature_by_group" max processes and threads was reported to be 5 and 7 respectively.

Normally I request from a cluster as many cores as parallel threads. But when you say that hmmsearch should be done in 8h on 4 cores, it sounds fast enough, optimisation may not be that much needed. I am checking now also a bit how VS2 scales with the size of the input data.

[jiarong]

@michalogit Thanks for showing the source report! I have something to learn from you. The rule hmmsearch has 6 wildcards in it. I thought snakemake submit each of the wildcards separately. But from the report, it seems that all wildcards on the rule hmmsearch is submitted together as one job (6 processes), or is this just the way snakemake summarize the rules by combining the stats from 6 separately submitted jobs? BTW, how many threads did you set for hmmsearch?

Another note is 8h walltime is highly dependent on FAA_BP_PER_SPLIT in template-config.yaml.