Unable to multi-thread MetaMaps (installation issue?)

[FabbriniMarco]

Hi everyone and thank you for the time you'll spend reading this issue.

I successfully installed MetaMaps on my server machine as suggested in the INSTALL.txt document, every step of the pipeline is working (db construction, mapping and classification, but i can't run any metamaps command on multithread, resulting in extremely long processing times. My machine of course allow for multithreading processes:

(base) [marco_f@server ~]$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 56 On-line CPU(s) list: 0-55 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz Stepping: 1 CPU MHz: 2899.951 CPU max MHz: 3300,0000 CPU min MHz: 1200,0000 BogoMIPS: 4800.05 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 35840K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts

I used the following command for read mapping and classification: metamaps mapDirectly --all -r /home/marco_f/databases/miniSeq+H/DB.fa -q input.fastq -o classified_output --maxmemory 200 --threads 30 metamaps classify --mappings classified_output --DB /home/marco_f/databases/miniSeq+H --threads 30

I successfully obtained output files with assigned taxonomy but looking at the processes with "top" the CPU usage was only 100% (1 thread). I tried running unset MALLOC_ARENA_MAX just before calling each of the command but nothing changes.

If it may be relevant, i installed boost and metamaps locally for my user conda base environment, since i don't have sudo permissions. I added boost lib directory as LD_LIBRAY_PATH to my bashrc as well as the BOOST_DIR.

Any suggestions?

Thank you for your time!

[tetukas]

Hi @AlexanderDilthey,

I'm experiencing exactly the same issue that was described by @FabbriniMarco regarding MetaMaps. Despite specifying more threads, the program seems to use just one. The attempts to solve this issue, including trying to run unset MALLOC_ARENA_MAX before each command, installing the program through different methods (conda, building from source), and even separately building boost libraries, haven't been successful.

Have you managed to find a solution or workaround to this issue? Any input would be greatly appreciated.

Thank you in advance!

[AlexanderDilthey]

Hi @FabbriniMarco and @tetukas,

I have unfortunately no particularly good ideas for how to debug this (and never encountered the issue myself) - I wonder whether the issue may have something to do with openMP.

If the issue is related to memory in some way, it may make sense to set up a reduced-size database and see whether the program then starts to use multiple threads.

It may also make sense to try a small thread count first, e.g. 2.

Best wishes

Alex