This repository contains the DPDK verbs source along with some demo applications.

Requirements

• Linux kernel >= 3.17.8
• gcc >= 4.9 or a compiler which supports the C11 standard
• rdma-core >= 17.1 (earlier versions, including the split libraries prior to the initial consolidated rdma-core release, are no longer supported)
• DPDK >= 16.07 built as shared libraries (CONFIG_RTE_BUILD_SHARED_LIB=y), and the rte_kni kernel module built for the currently running kernel
• libnl-3 and libnl-route-3
• json-c
• uthash (included in Fedora, Ubuntu, openSUSE, and EPEL for RHEL repositories)

Setup instructions

I am assuming a default installation of Ubuntu 18.04. Other modern distributions should also work, but you would need to install DPDK yourself in order to build the required rte_kni kernel modules which most other distributions do not ship in their repositories.

To build this package, RTE_SDK and RTE_TARGET need to be exported into the environment. Using the packages in the Ubuntu 16.10 repositories, you can run the following:

$ source /usr/share/dpdk/dpdk-sdk-env.sh

Or for a manual DPDK installation:

$ export RTE_SDK=${prefix}/share/dpdk
$ export RTE_TARGET=x86_64-native-linuxapp-gcc

If you are pulling this from a fresh git clone, first run:

$ autoreconf -i

Then this follows the normal autotools-style build:

$ ./configure --sysconfdir /etc
$ make
$ sudo make install

Note that sysconfdir must match that of your libibverbs installation, in order for the verbs library to find the urdma driver.

The configure script will look for your kernel source directory in the typical location by default (/lib/modules/uname -r/source). You can set the KERNELDIR environment variable to specify a different location; for example, if you are building against a different kernel version than what it installed locally.

To run an application with this driver, the KNI and urdma modules must be loaded:

$ sudo modprobe rte_kni
$ sudo modprobe urdma

You need to use the dpdk-devbind script to bind the NICs that you wish to use with rdma to a DPDK-compatible UIO/VFIO kernel driver, instead of the default kernel netdev driver. Refer to the DPDK documentation for more information on this.

You will need to create a file ${sysconfdir}/rdma/urdma.json that looks like this, with appropriate values substituted in:

{ "ports": [ { "ipv4_address": "10.2.0.100" } ],
  "socket": "/run/urdma.sock",
  "eal_args": { "log-level": 7 }
}

You can validate your config file against doc/urdma-schema.json using any JSON schema validator; python-jsonschema which comes with Ubuntu is one possibility. Note that this schema file is stricter than what is actually allowed at runtime; at runtime, additional properties are simply ignored but the schema file does not allow them; this is to make typos more obvious.

Finally, the urdmad service must be running:

$ systemctl --user start urdmad

This will cause devices to appear in your "ip link" output, and cause uverbs devices to appear in /sys/class/infiniband_verbs.

Or you can manually run urdmad as root; in this case you will need to run every verbs application as root to pick up the DPDK configuration which cannot be relocated.

Troubleshooting Notes

Some RDMA demos, particularly the tools that come with librdmacm, can give confusing error messages when there are no RDMA devices available, along the lines of:

rdma_create_event_channel: No such device

If you see this error message, it indicates that urdma is not set up properly. Verify that you followed the steps above properly.

Known Issues

• Shared receive queues (SRQs) are currently not supported. In order to run openmpi over urdma, you will need to specify the following command line options to prevent openmpi from using shared receive queues, and to disable a warning that it doesn't know about the device vendor ID:

  $ mpirun --mca btl_openib_warn_no_device_params_found 0 \ --mca btl_openib_receive_queues P,65536,256,192,128 \ ${mpi_app} ${mpi_app_args}...

• If running DPDK sample applications succeeds, but running urdmad fails with "Configuration expects N devices but found only 0", the DPDK PMD drivers are probably not being loaded at runtime by default. If this is the case then you will probably need to load them by hand, by adding an argument to urdmad like:

  urdmad -d ${RTE_SDK}/${RTE_TARGET}/lib/libpmd_net_i40e.so.1

  and adding the equivalent to urdma.json:

  "eal_args": { / ... /, "d": "..." }

  To avoid this, you can set CONFIG_RTE_EAL_PMD_PATH to a directory like /usr/local/lib/dpdk-pmds when building DPDK, and the place the PMD .so files into that directory after DPDK is installed. DPDK will then auto-load all .so files in that directory as PMD libraries.

• DPDK 17.05 introduced the concept of mempool drivers. If you see a message like this using DPDK >= 17.05:

  MBUF: error setting mempool handler EAL: Error - exiting with code: 1 Cause: Cannot create rx mempool for port 0 with 8064 mbufs: Invalid argument

  then you do not have a mempool driver by default. This occurs because the linker has been configured with --as-needed by default on some distributions, and since the mempool and PMD drivers do not declare any symbols, the linker has no way of knowing that we depend on the presence of a mempool driver.

  Like the PMD issue above, you can pass -d to load the default librte_mempool_ring driver on the command line, or set CONFIG_RTE_EAL_PMD_PATH and place mempool driver(s) into that directory.

• There is a potential race condition with completion channels, where a completion event can get lost, and thus a thread waiting on ibv_get_cq_event() will never wake up, leading to a deadlock. A cause has not been identified, but the issue has not been reproduced with the "extra" lock added around the rte_ring operations in do_poll_cq() and finish_post_cqe().

• There is the possibility of a hang in the kernel module if the user process is killed while between the read() and write() calls on event_fd in poll_conn_state(). This is because rdma_destroy_id() in the kernel will block until the connection attempt completes, but itself prevents our event_fd from being closed which would unblock it.

• The progress thread will use 100% CPU since it must busy-poll on the KNI interfaces (there is no way to sleep until the process gets an event).

• urdma follows the RFC 5040 ordering rules strictly, meaning that it can place data segments out of order if it receives them out of order. This in turn means that if two RDMA WRITE requests are made on overlapping buffers, urdma may place a data segment from the first after the corresponding data segment from the second, thus leading to a torn write from the perspective of the application. Thus applications must not post multiple transfer requests on overlapping buffers simultaneously if they depend on the data ordering.