Open YoniSp opened 2 years ago
Those errors can be ignored. Sometimes CUDA calls return an error, and it's okay, NCCL ignores the error and continues because that error is not a problem. In the case above, the first error is because we try to dynamically load optional functions and we don't find one (not a problem), and the second is because we try to enable p2p to a peer which we already connected to before (again, not a problem, we just want to make sure P2P is enabled to that peer).
During running the follwing example with sanitizer: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/examples.html#example-1-single-process-single-thread-multiple-devices I am facing the next error:
I am compile the project using cmake
set(CMAKE_CUDA_ARCHITECTURES 86 80) set(CMAKE_CUDA_COMPILER "$ENV{NVHPC_PATH}//cuda/bin/nvcc") set(CMAKE_CUDA_COMPILER_FORCED TRUE) project(Test C CXX CUDA) set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH} "$ENV{NVHPC_PATH}/cmake" ) find_package(NVHPC REQUIRED ) ADD_EXECUTABLE(NcclToySingleThreadMultiGpu NcclToySingleThreadMultiGpu.cu) target_link_directories(NcclToySingleThreadMultiGpu PUBLIC ${NVHPC_CUDA_LIBRARY_DIR}) TARGET_LINK_LIBRARIES(NcclToySingleThreadMultiGpu NVHPC::NCCL NVHPC::CUDA muparser pthread)
My setup is: GPU rtx3090 + nvlink NVHPC 22.7 OS Rocky 8what am I doing wrong and how can I over come the issue?