Open johnmauff opened 1 year ago
With the most recent update of system software on Gust the cuIpcGetMemHandle error appears to have been eliminated.
Thanks for the update!
I am seeing the same issue, currently, with just the simple OSU benchmarks. So we are tracking it with HPE.
Glad you are unaffected now.
@benkirk I just realized that the error went away for me due to the removal of management memory in the makefile, and not anything with the new system software. I will discuss this with my NVIDIA contacts to see if they have any ideas.
After setting the CUDA_VISIBLE_DEVICES environment variable #22 I was able to launch CM1 on multiple nodes of Gust. Unfortunately, it subsequently dies with the following error before the timestep is entered. Note that while CM1 is an OpenACC code, it does not use OpenACC to access the MPI library. Note that a similar failure was observed on Perlmutter by Supreeth Suresh and may be related to the Cray MPICH library.
(GTL DEBUG: 1) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 97