Open devreal opened 1 year ago
I think this is similar to an issue we reported here - https://github.com/openucx/ucx/issues/8761
You can try setting UCX_MEM_CUDA_HOOK_MODE=reloc
as a workaround. The bistro
cuda hook is missing some functionality for powerpc.
Thanks @nysal. I can confirm that setting UCX_MEM_CUDA_HOOK_MODE=reloc
yields good latencies. I suggest keeping this issue open until https://github.com/openucx/ucx/issues/8761 has been resolved, in case others come across this.
Background information
What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)
Open MPI main branch (0ba5d47dc49ca333f99fb9e90e213191acad667b)
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Installed with CUDA support (11.0.3 - the default on Summit) and UCX 1.14.0.
Loaded modules:
UCX config summary:
If you are building/installing from a git clone, please copy-n-paste the output from
git submodule status
.Please describe the system on which you are running
Details of the problem
I am seeing high latencies in the osu_latency benchmark (OSU benchmarks 7.0.1) when using
pml/ucx
:If I disable
UCX_IB_GPU_DIRECT_RDMA
the latency is slightly lower, but still too high:With
pml/ob1
andbtl/uct
the latency is reasonable but I get a Segfault on messages above 4KB:I repeated the experiment with CUDA 11.5.2 (the newest available module on Summit). The latencies are similar, but with pml/ucx I get this warning:
What am I doing wrong here? Anything I should set to get latencies that are closer to those I get with host memory?