mpirun + hyperthreading

[pmav99]

I am running on a laptop with an AMD Ryzen 4800H processor. This means that I am using an 8 cores/16 threads machine.

When I try to execute launchshism.sh I get the following error:

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 15
slots that were requested by the application:

  schism

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------

My understanding is that the number of cores that is being passed to the CPU is determined by using multiprocessing.cpu_count().

Side note: according to the docs, len(os.sched_getaffinity(0)) is probably a better choice in the general case, but that's not important here. Both methods return 16 on my setup.

Anyhow, even though 15 cores/threads are indeed available, nevertheless, mpirun throws the error I pasted. I googled a bit and I found that there are two options to resolve this:

1. Use mpirun --oversubscribe ... which in case you do oversubscribe the machine, might degrade performance
2. Use mpirun --use-hwthread-cpus ... which AFAI can tell allows mpirun to use CPU threads as cores.

Option 2 feels like the better choice, but I don't have much experience with MPI so I can't really say if there are any downsides.

[brey]

I think --oversubscribe will just load the cpus. I am not sure whether the compiled version of SCHISM can indeed recognise the threads. I believe this is an openMP feature that SCHISM (although implemented partially) doesn't support fully.

Did you compile SCHISM locally or use the conda version?

Are you using openMPI (different than openMP) or MPICH (yet another version of MPI)? Note that there two different conda version based on MPICH or openMPI.

There are a lot of issues here that needs to be clarified before trying for maximum efficiency. In fact, some of these issues are not yet clear to me.

[brey]

I think openMPI & MPICH support multi-threading.

https://princetonuniversity.github.io/PUbootcamp/sessions/parallel-programming/Intro_PP_bootcamp_2018.pdf

[pmav99]

I just installed using conda: conda install -c gbrey pschism.

$ mpirun -V
mpirun (Open MPI) 4.0.5

$ which mpirun
/home/panos/.conda/envs/pyposeidon/bin/mpirun

The problem is that multiprocess.cpu_count() returns 16 and that mpirun is not recognizing/accepting the CPU threads as cores. Therefore, when the tests get launched there are failures because the generated launchschism.sh scripts contain the line mpirun -N 15 schism.

If I manually edit launchschism.sh to reduce the number of CPUs to 8 or add --use-hwthread-cpus then schism does run.

The funny thing is that on my old laptop which used an (older) intel CPU with 4c/8t (i7 6700HQ if memory serves) there was no such issue. So perhaps this issue only affects MPI + Ryzen...

[pmav99]

If we are not sure about the mpirun options, perhaps a valid workaround would be to ignore the number of threads and instead use the number of physical cores as the default value. The latter can be determined by using e.g.:

import psutil

NCORES = psutil.cpu_count(logical=False)
NTHREADS = psutil.cpu_count(logical=True)

(source)

This would make the tests run a bit slower but I guess that they would reliably run on all CPUs regardless of number of sockets/hyperthreading etc. We could add a configuration option that would allow to switch to logical cores, too.

[brey]

We are already usingmultiprocessing.cpu_count() in pyposeidon or not?

[pmav99]

Yes we do. E.g. https://github.com/ec-jrc/pyPoseidon/blob/06cdcaa21f839f8531e067e9c633cc292fdbe30f/pyposeidon/schism.py#L51

Just to make this more clear:

import multiprocessing
import psutil

print(f"Multiprocessing      : {multiprocessing.cpu_count()}")
print(f"psutil logical cores : {psutil.cpu_count(logical=True)}")
print(f"psutil physical cores: {psutil.cpu_count(logical=False)}")

On my laptop, the above snippet results in:

Multiprocessing      : 16
psutil logical cores : 16
psutil physical cores: 8

[brey]

On my mac

Multiprocessing : 8 psutil logical cores : 8 psutil physical cores: 4

but running with 7/8 cores poses no problem.

[pmav99]

According to both the man pages of mpirun and this SO thread, the correct approach appears to be the --use-hwthread-cpus flag. Enabling the flag should not lead to oversubscription. It should just allow MPI to discover the proper number of "hardware threads".

What is of interest is the last sentence of the first answer in the SO thread:

MPI tasks are not bound on cores nor threads on OS X, so if you are running on a Mac, the --oversubscribe -np 4 would lead to the same result.

so it might be that OS X behaves differently than Linux

[brey]

I just got the same warning with openmpi on my mac. Mpich has no issues. The --use-hwthread-cpus worked.