We perform a lot of small processes (mafft, fasttree, nw_display) that not necessarily need to run in a single job submission when executing the pipeline on an HPC or Cloud.
Solution:
cluster processes together in chunks of e.g. 20 or 50 files and then submit jobs. So instead of submitting 1000 mafft jobs submit 20*50 mafft jobs.
We perform a lot of small processes (mafft, fasttree, nw_display) that not necessarily need to run in a single job submission when executing the pipeline on an HPC or Cloud.
Solution: cluster processes together in chunks of e.g. 20 or 50 files and then submit jobs. So instead of submitting 1000 mafft jobs submit 20*50 mafft jobs.
This should help with latency problems on HPCs.