Benchmark data for HPC cluster

[fpeti11]

I would like to ask if you have some benchmark data for HPC clusters that proves the program can use HPC resources in good way?

[fpusan]

We don't. My limited experience with this is that:

• Most time consuming steps will make use of all the threads you request, so efficiency is probably decent in that regard.
• Assembly and annotation are memory consuming, other steps not so much (LCA and mapping may also consume lots of memory when using lots of threads, but this has improved recently). So in a way the memory you request is being "wasted" during the rest of the steps.
• Steps that need to read from the reference database (annotation and LCA) may have IO bottlenecks depending on where the databases are located. Ideally you'd want them in a place with high transfer speeds / low latency.

The different steps of the pipeline can be run individually. You can create a skeleton project by adding the flag --empty to the SqueezeMeta call. After that, the different scripts of the pipeline can be run individually (eg 01.run_all_assemblies.pl my_project_path). So you could as well submit a different job for each step, optimizing the resources requested for each job.

Hope this helps a bit.

[fpeti11]

Thank you, it helped.