franciscozorrilla
commented
5 months ago Hi Kunal,
There are sections of code in the crossmap rule that probably do not make full use of parallelisation, e.g. when combining multiple sorted bam files with jgi_summarize_bam_contig_depths or concoct_coverage_table.py, although there is not much I can do about this. I would suggest that you monitor your job with htop to see if at least the bwa-mem commands are getting multiples cores. Also, it is possible that you may have properly assigned your cores in the config.yaml file (used for workflow parameters) but NOT in the cluster_config.json file, which is the one that actually requests resources from your cluster. Please double check these files.
Also how large is your dataset? For e.g. more than 250 samples it becomes difficult to obtain cross mapping results in a reasonable amount of time. By default metaGEM runs crossmapping in series (i.e. one job per focal sample, then each job maps short read samples to the focal sample assembly one at a time), have you considered the cross mapping in parallel? However, again for large datasets this will spawn a huge number of jobs that could take a while to finish on the cluster. If I were you, I would use the default crossmapping in series approach, but instead of bwa-mem I would use this new fairy tool that just came out that is apparently ~250x faster. Seems easy to use and is compatible with binning tools, here is the preprint and github repo if you are interested. I will definitely replace bwa-mem with this tool in the metaGEM workflow once I have some time in the future, but in the meantime please have a look.
Best, Francisco
kunaljaani
Hi Francisco,
I am running the metaGEM locally on the workstation with 128 cores and 512 RAM. I changed the available thread in the config file to the maximum. But while running crossmap step it seems to use limited threads (4/5) which adds to the longer processing time. Could you please suggest some alternative that will allow the crossmap step to run faster?
Thank you Kunal