search

ntrdma / ntrdma-ext

Linux extra (out of tree) kernel modules for ntrdma.
Other
20 stars 11 forks source link

Setting up ntrdma #11

Closed starnullptr closed 4 years ago

starnullptr commented 4 years ago

I was trying to set up ntrdma but couldn't find steps to install and run it on two machines.

I am following https://github.com/ntrdma/ntrdma-ext.

1) My kernel version is 5.2.14. 2) I was able to compile using YADRO-KNS:linux-5.2.x-to-upstream. However I am not able to follow what I need to do next. I tried qperf example from https://github.com/ntrdma/ntrdma-ext but got "failed to find any InfiniBand devices ".

What am I missing?

alfonder commented 4 years ago

@anubhavguleria47 what hardware do you use as NTB?

starnullptr commented 4 years ago

Hi @alfonder , Sorry for late response. I am using Intel NTBs connected in b2b configuration.

alfonder commented 4 years ago

Hi, @anubhavguleria47. PR from "YADRO-KNS:linux-5.2.x-to-upstream" branch was rejected as it used the old code base of "ntrdma-ext" project. That branch is discontinued. Now I'm preparing a new PR, which will be based on the current snapshot of "ntrdma-ext". And it will bring Linux 5.5.x support.

starnullptr commented 4 years ago

@alfonder okay. Also, while using ntrdma, the rdma-core I need to use is https://github.com/ntrdma/rdma-core ? And what steps do I need to follow after loading ntrdma driver? Currently I see eth0 and I manually assign ip to it on both the machines. Is there something else also required?

starnullptr commented 4 years ago

@alfonder what kernel version should I use with the current ntrdma-ext ?

alfonder commented 4 years ago

@anubhavguleria47 currently ntrdma-ext supports linux 4.14 AFAIK.

alfonder commented 4 years ago

And what steps do I need to follow after loading ntrdma driver? Currently I see eth0 and I manually assign ip to it on both the machines. Is there something else also required?

I think you should be able to run tests from README after your virtual network interface is working. Personally I can't check this as far as my hardware is not supported in version 4.14

starnullptr commented 4 years ago

Hi @alfonder, once I built and loaded ntrdma kernel module, do I need to enable some specific setting to make ntrdma_0 work with rdma-core? Here is the response for ./ibv_devices and ./ibv_devinfo that I get:

./ibv_devices device node GUID

ntrdma_0 0000000000000000

./ibv_devinfo Failed to open device It fails to open /dev/infiniband/uverbs0 . Do I need to add/modify any udev rule? I have loaded ib_uverbs and running everything with sudo.

starnullptr commented 4 years ago

@alfonder I was able to get it work. Thanks.