how to manually enable parallelization in hpc environment?

[jfx319]

When running mixcr on a local machine (e.g. laptop), the program automatically uses all cores.

But when running on a cluster (e.g. with slurm scheduler), it is single-threaded and ignores the extra cores reserved.

Is there a more explicit way to manually specify parallelization & memory, rather than relying on presumably an auto-detect mechanism (that seems to fail in this HPC environment)? It's also difficult to use auto-detect on a shared node in HPC, since potentially the usual commands to check max mem/cores reveals more resources (the whole node) than actually reserved for this particular job (a fraction of the node).

Any advice would be appreciated!

[dbolotin]

Hi,

You can use:

• -t option on all analysis steps that supports parallelization to override the auto-detected concurrency level
• -Xmx option to specify amount of memory available to the process

e.g.

mixcr align -t 4 -Xmx4g ...

MiXCR uses some empirical formulas to determine the amount of available memory (see script) and [availableProcessors()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Runtime.html#availableProcessors()) method to determine number of CPUs.

[jfx319]

Thanks!

Is there a way to pass such flags to the main wrapper? Or does each step need to be deconstructed?

mixcr -t 4 -Xmx16g analyze shotgun ... etc

If diving into specific steps is required, where can I find a vignette tutorial workflow that might help with understanding how to use each step and how to interpret the results of each step in the context of a real dataset?

[dbolotin]

mixcr analyze will correctly pass both options to all the steps.

[jfx319]

Minimal reproducible example:

mkdir test
cd test
wget https://github.com/milaboratory/milib/raw/f9219a0b31e37ec45c3cdb055bd1e8463499a3ff/src/test/resources/sequences/sample_r1.fastq.gz
wget https://github.com/milaboratory/milib/raw/f9219a0b31e37ec45c3cdb055bd1e8463499a3ff/src/test/resources/sequences/sample_r2.fastq.gz

#wasn't sure about species or rna vs dna (feel free to change)
mixcr -t 4 -Xmx4g analyze shotgun \
     --species HomoSapiens \
     --starting-material dna \
     --receptor-type xcr \
     sample_r1.fastq.gz \
     sample_r2.fastq.gz \
     test

ERROR:

Unknown options: -t, 4

looks like the -Xmx4g flag works, but not the -t flag; am I invoking it correctly?

Version info:

mixcr -v

MiXCR v3.0.12 (built Wed Nov 20 08:38:34 EST 2019; rev=cad6564; branch=release/v3.0.12; host=Dmitrys-MacBook-Pro.local)
RepSeq.IO v1.3.3 (rev=fd1bd59)
MiLib v1.12 (rev=12b1449)
Built-in V/D/J/C library: repseqio.v1.6

[dbolotin]

I'm really sorry, I misinformed you. I remembered that we implemented it, but in fact we didn't.

You should specify -t option for each of the steps, like this:

mixcr -Xmx4g analyze --align "-t 4" ...

-Xmx4g should be specified only once.

[dbolotin]

Just created an issue for this #563.

[jfx319]

No problem, thanks for your help on this!

I will use the option you suggested in the mean time. Is there another step besides "align" that you think is worth specifying more threads to? On an initial test, I recall the align step seemed to take the vast majority of the time, but am not sure if another step (e.g. contig assembly) would benefit too. Looking through readthedocs, I only see --assemble having the option. Let me know if I'm missing any.

I am guessing it would be:

numThreads=4

mixcr -Xmx4g analyze shotgun \
    --align "-t ${numThreads}" \
    --assemble "-t ${numThreads}" \
    --contig-assembly \
    --species HomoSapiens \
    --starting-material dna \
    --receptor-type xcr \
    sample_r1.fastq.gz \
    sample_r2.fastq.gz \
    test

I tested this and it seems to work fine; more importantly, it now substantially decreases the predicted align ETA to ~34 hours for a WGS with 64 threads.

[PoslavskySV]

You should also add -t option for contig-assembly (--contig-assembly "-t ${numThreads}").

We've fixed the problem in #563 and will include it in MiXCR 3.0.13 which will be released on the next week. We'll let you know. Thanks for reporting!

[jfx319]

Thanks!

You should also add -t option for contig-assembly (--contig-assembly "-t ${numThreads}").

I tested this and it complains that there is no -t option (removing this from --contig-assembly fixes the complaint).

[PoslavskySV]

Fixed in 3.0.13, release on road

[github-wow]

@jfx319 you wrote:

"When running mixcr on a... cluster (e.g. with slurm scheduler), it is single-threaded and ignores the extra cores reserved."

Can you please suggest how did you check that?