Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
************************************
* Waiting for client to connect... *
************************************
initializing CUDA
Listing all CUDA devices in system:
CUDA device 0: PCIe address is 1B:00
CUDA device 1: PCIe address is 3E:00
CUDA device 2: PCIe address is 89:00
CUDA device 3: PCIe address is B2:00
Picking device No. 0
[pid = 3164333, dev = 0] device name = [NVIDIA GeForce RTX 4090]
creating CUDA Ctx
making it the current CUDA Ctx
CUDA device integrated: 0
cuMemAlloc() of a 131072 bytes GPU buffer
allocated GPU buffer address at 00007c43eac00000 pointer=0x7c43eac00000
Couldn't allocate MR
failed to create mr
Failed to create MR
Couldn't create IB resources
destroying current CUDA Ctx
./ib_send_bw --use_cuda=0 192.168.2.244
Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
initializing CUDA
Listing all CUDA devices in system:
CUDA device 0: PCIe address is 1B:00
CUDA device 1: PCIe address is 3E:00
CUDA device 2: PCIe address is 89:00
CUDA device 3: PCIe address is B2:00
Picking device No. 0
[pid = 3164350, dev = 0] device name = [NVIDIA GeForce RTX 4090]
creating CUDA Ctx
making it the current CUDA Ctx
CUDA device integrated: 0
cuMemAlloc() of a 131072 bytes GPU buffer
allocated GPU buffer address at 00007847fac00000 pointer=0x7847fac00000
Couldn't allocate MR
failed to create mr
Failed to create MR
Couldn't create IB resources
destroying current CUDA Ctx
If GPUDirect is not working, (e.g. you see "Couldn't allocate MR" error message), consider disabling Scatter to CQE feature. Set the environmental variable MLX5_SCATTER_TO_CQE=0. E.g.:
MLX5_SCATTER_TO_CQE=0 ./ib_write_bw -d ib_dev --use_cuda= -a
system info
nvidia driver and cuda version:
peermem module status:
error occured:
./ib_send_bw --use_cuda=0
./ib_send_bw --use_cuda=0 192.168.2.244