search

Mellanox / nv_peer_memory

305 stars 61 forks source link

If connectX-3 HCA can use ibv_reg_mr to register GPU memory created by cudaMalloc? #36

Closed YuxinxinChen closed 6 years ago

YuxinxinChen commented 6 years ago 
yuxin420@luigi:~/TEMP/rdma$ lspci -v | grep Mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]
82:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]

I installed Mellanox OFED to use IB verbs. It works fine if the buffer is on CPU memory. But if I ibv_reg_mr to register a buff on GPU, it fails. If I need to use another verb for registering GPU memory or I should do something setting stuff to enable it?

Thanks!

Yuxin

ferasd commented 6 years ago

Is nv_peer_mem module loaded? if so, ibv_reg_mr should work fine.

ferasd commented 6 years ago

@YuxinxinChen

YuxinxinChen commented 6 years ago

Thank you very much. Problem solved!

EsdeathYZH commented 3 years ago

I encounter the same problem, but the nv_peer_mem module has been loaded in my kernel successfully, how do you solve the problem finally?@YuxinxinChen