Use fewer threads for small alignment

[alexweisberg]

Hi, raxml-ng is fantastic and I am using it as a replacement for standard RAxML in many cases.

I am running into an issue regarding the number of threads. Occasionally with a small alignment I am getting the error:

ERROR: Too few patterns per thread! RAxML-NG will terminate now to avoid wasting resources. NOTE: Please reduce the number of threads (see guidelines above). NOTE: This check can be disabled with the '--force' option.

I understand the force option would ignore this, however I would prefer not to ignore other errors, and it would be convenient if raxml-ng instead just used fewer threads and treated the --threads option as a "max threads" option instead.

[amkozlov]

Hi Alexandra,

thanks for your positive feedback!

Would it help if I add a new

--threads auto

option with the meaning

MIN(auto-detected number of CPU cores, number of threads which can be efficiently used)?

I know this error might be annoying, but I would like to make sure that users don't run raxml-ng with 1-2 threads on a 16-core server because their alignment is short (or at least if they do, it's their informed decision).

[alexweisberg]

That would be great! As long as it used something like OMP_NUM_THREADS or some other way to limit it to a max number of threads, that way it could be limited when run on a cluster machine with more cpus but with other jobs running.

[amkozlov]

ok I see your point, then it should really be something like --max-threads (raxml-ng doesn't use OpenMP,, but I can still read OMP_NUM_THREADS value of course, I'll give it a thought)

[alexweisberg]

Sure, the value was just an example. Maybe something like --threads auto{#} similar to how pars{#} and rand{#} work for starting trees?

[amkozlov]

that actually looks very neat, thanks for the suggestion!

[tseemann]

I would love to see --threads auto too.

iqtree used to have this same issue. It now has -ntmax 36 -nt auto type syntax

[amkozlov]

@tseemann thanks for your feedback! I didn't get to this yet, but it's on my list.

Meanwhile, there are two alternative solutions/workarounds:

1. "Too many threads" error can be disabled individually with --force perf_threads, without affecting other safety checks.
2. On short alignments, we can utilize more cores efficiently by parallelizing across tree searches/bootstrap replicates. This coarse-grained parallelization feature is still experimental and is available in a separate branch, but it will be integrated into the next raxml-ng release. More on coarse-grained parallelization here: https://github.com/amkozlov/raxml-ng/wiki/Parallelization#coarse-grained-parallelization-for-short-alignments

[amkozlov]

This is now implemented in coarse branch and will be available in upcoming raxml-ng 1.0 as part of parallelization autoconfig.

See c28c1e1 for details.