Closed benkirk closed 1 year ago
So this came about after Jeremy reported needing to specify MPICH_GPU_SUPPORT_ENABLED 1
in his batch jobs to use GPUs with cray-mpich. I figured adding it to the cray-mpich module would be a convenience to users as online documentation suggests it should only potentially add a small launch delay, but obviously it can't be set for applications built without GPU hooks.
I believe we need to add this variable to our documentation. I'll take a pass at that.
Added the following text to our draft doc:
If you are using an MPI application compiled with GPU support, you will need to set/export the environment variable MPICH_GPU_SUPPORT_ENABLED=1 before calling the MPI launcher in your job. This variable, when active, tells cray-mpich to enable CUDA functionality.
Longer term on Derecho, do you think it's worth looking at a PBS hook or mpiexec wrapper that sets this automatically only when GPUs are being requested?
I think it is worth exploring. Only potential problem I see is if they try to run a non-GPU MPI app in the same job. In some cases we probably want that to break, but maybe there are legit use cases?
Resolved by inclusion into the get_local_rank
helper script.
Under
ncarenv/22.08b
:Reverting to
ncarenv/22.08
reveals that stack is unaffected.