oversubscription during rmblastn

[nathanweeks]

RepeatModeler accepts a -pa # option to set the number of child processes it will fork, including for rmblastn tasks. However, RepeatMasker::NCBIBlastSearchEngine by default invokes rmblast with -num_threads 4, e.g., leading to a 4X oversubscription of rmblast threads to processors if the user specifies -pa $(getconf _NPROCESSORS_ONLN).

Perhaps beforehand, RepeatMasker::NCBIBlastSearchEngine->setCores(1); could be used, or maybe instead if using a multi-threaded rmblast, then simply RepeatMasker::NCBIBlastSearchEngine->setCores($options{'pa'}), and not fork any child processes?

[jebrosen]

At the very least there is a documentation mismatch. The short description of -pa says "Number of processors to use in parallel" but the longer help states "RepeatMasker uses 4 PIDs for each parallel job". In actuality, -pa controls the number of simultaneous search jobs, each one using up to 4 CPUs depending on the search engine.

[jebrosen]

I misread this as a question about RepeatMasker. The help text I referred to is from RepeatMasker not RepeatModeler, but the situation is the same.

[nathanweeks]

FWIW, RepeatModeler -h says:

    -pa #
        Specify the number of shared-memory processors available to this
        program. RepeatModeler will use the processors to run BLAST searches
        in parallel. i.e on a machine with 10 cores one might use 1 core for
        the script and 9 cores for the BLAST searches by running with "-pa
        9".

[rmhubley]

We will correct the docs. Due to differences in efficiency between search engines we chose to divorce the batch parallelization from the core usage of the search engine in question. I can see where this could lead to problems with jobs schedulers in a clustered environment. For others reading this issue, here are the default cores used by each invocation of a search engine:

RMBlast 4 nhmmer 2 cross_match 1

To calculate the cores used by RepeatMasker with "-engine nhmmer -pa 4" it would be 4 (batches) * 2 (cores) + 1 (perl main process) or 9 cores in total.

[xiaoyezao]

Sorry to open this issue again. But I used -pa 6 on a 24-core cluster, there seems only one rmblastn job running with only 1 core (observed in round-6).

[jebrosen]

@xiaoyezao It is possible that work was not divided exactly evenly, so only one thread was left running by the end. If you can reliably find that only 1 job or core is running at a time throughout the run (not only at the end), please open a new issue with more details about your computing environment. I think that would be the first time we have heard of such a problem.

[rmhubley]

The current version of RepeatModeler has removed the "-pa" parameter and switched to a parameter ('threads') that guarantees that at most a particular number of threads will be used by the program.