search

SoftRoCE / rxe-dev

Development Repository for RXE
Other
132 stars 56 forks source link

ibv_devices showing no rxe0 or GUID #75

Closed fwmiller closed 6 years ago

fwmiller commented 6 years ago

I think this is similar to another issue #48 I believe.

I've got a good build now that comes up with SoftRoCE and seems to partially work. I'm trying to do an rping between the Arria10 and a PC VM. Here's some info. I've loaded all the Infiniband stuff as modules:

root@arria10:/# lsmod Module Size Used by rdma_rxe 102400 0 rdma_ucm 28672 0 rdma_cm 53248 1 rdma_ucm iw_cm 32768 1 rdma_cm ib_cm 40960 1 rdma_cm ib_uverbs 81920 1 rdma_ucm ib_core 192512 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm

I can see the rxe_cfg stuff:

root@arria10:/# rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU eth0 yes st_gmac 1500 10.0.1.25

However, the RMTU is missing here and when I execute ibv_devices, I don't get an rxe0 device with a GUID like I should.

root@arria10:/# ibv_devices device node GUID

Do you know about why this would be? Is something missing?

Thanks, FM

fwmiller commented 6 years ago

I changed the order of my commands and I get different output. I'm getting this warning about not supporting the kerne ABI of 1:

root@arria10:~# rxe_cfg
rdma_rxe module not loaded
  Name  Link  Driver   Speed  NMTU  IPv4_addr  RDEV  RMTU
  eth0  yes   st_gmac         1500  10.0.1.26
root@arria10:~# rxe_cfg start
[  125.217418] rdma_rxe: loaded
[  125.247695] rdma_rxe: set rxe0 active
[  125.251445] rdma_rxe: added rxe0 to eth0
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
  Name  Link  Driver   Speed  NMTU  IPv4_addr  RDEV  RMTU
  eth0  yes   st_gmac         1500  10.0.1.26  rxe0  (?)
root@arria10:~# rxe_cfg add eth0
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
root@arria10:~# rxe_cfg start
[  141.360121] rdma_rxe: already configured on eth0
sh: line 0: echo: write error: Invalid argument
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
  Name  Link  Driver   Speed  NMTU  IPv4_addr  RDEV  RMTU
  eth0  yes   st_gmac         1500  10.0.1.26  rxe0  (?)
root@arria10:~# rxe_cfg
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
IB device 'rxe0' wasn't found
  Name  Link  Driver   Speed  NMTU  IPv4_addr  RDEV  RMTU
  eth0  yes   st_gmac         1500  10.0.1.26  rxe0  (?)
root@arria10:~# ibv_devices
libibverbs: Warning: Driver rxe does not support the kernel ABI of 1 (supports 2 to 2) for device /sys/class/infiniband/rxe0
    device                 node GUID
    ------              ---------------
fwmiller commented 6 years ago

Moving to kernel 4.17 with the most recent rdma-core sucked into the yocto build fixed this problem

akifsaglam commented 5 years ago

I ran into this issue with exactly the same observation as above. I am using 16. 04 ubuntu. All you need to do is to add location of libibverbs.so to LD_LIBRARY_PATHS

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:path_to_softroce/rdma-core/build/lib