metONTIIME Running Slow - Cannot Increase RAM

[danuccio]

I have set metONTIIME to use 200 GB of RAM on a device with ~250GB for the more RAM intensive steps (e.g. taxonomic assignment).

withName:assignTaxonomy{
            container = 'maestsi/metontiime:latest'
            cpus = { params.assignTaxonomy ? 6 : 1 }
            memory = { params.assignTaxonomy ? 200.GB + (2.GB * (task.attempt-1)) : 1.GB }
            errorStrategy = { task.exitStatus == 130 ? 'retry' : 'terminate' }
            maxRetries = 3

However, whenever I check htop, metONTIIME never seems to be using more than ~6-20GB or RAM.

At present, my current run has been going on for nearly one month. It has been performing the taxonomic assignment step for the past three weeks.

While troubleshooting an earlier step, I reached out to one of the developers on this page about increasing CPUs in addition to memory, but was advised against doing so.

https://github.com/MaestSi/MetONTIIME/issues/100

Based on the parameters I included above, can you tell if I have this set up properly?

Do you have any additional recommendations?

Thank you

[MaestSi]

Hi, in the previous issue you raised, the program crashed due to unavailable RAM memory. In this run, it seems it is not crashing (at least, up to now), but of course it is taking a long time. I think it's a matter of finding the right compromise between number of threads (the higher the faster the process will be, but also it will require more memory) and consumed memory (here you have a hard limit, which may the process crash). My advice would be to set downsampleFastq process to true (if it is not already set) and to set a lower maxNumReads, a few tens of thousands per sample should be ok, but I don't know how many samples you are analysing. If you have many samples, you should consider splitting them in groups, analysing each group separately and then merging feature tables afterwards. Please also check that you are editing the portion of the config file related to the profile (i.e. docker or singularity) you are using. Best, SM