Question/Request-for-Information on configuring DPDK NICs for use with eRPC

[gshanemiller]

I'm trying to use eRPC on AWS .metal instances. I have the code built with AWS vfio-patches. In order to be logged into the machine while having a free ENA NIC for eRPC, I attach a second NIC. ./dpdk-devbind.py is able to see the second NIC, bind to it. All that's working fine.

Here's my question: the second ENA NIC which is bound to vfio-pci doesn't have an IP address e.g. running ip a does not list the second NIC as it's bound to vfio-pci for direct access. On the other hand the hello world needs a server name / IP-address for the client and sever. What should I provide there? My choices seem to be two. Which one is correct?

• I provide the IP addresses of the first NIC, and somehow/someway DPDK+eRPC finds the second NIC bound to vfio-pci on the client and server machines

  • (or) Before or after binding the second NIC to vfio-pci I must run some additional commands to associate an IP address with it and it's these addresses I provide in the hello world code for client/server

[anujkaliaiitd]

Hello. As of now, I've not tested eRPC on AWS though it might "just work". eRPC is supported on Azure (see https://github.com/erpc-io/eRPC#running-erpc-over-dpdk-on-microsoft-azure-vms).

The IP addresses provided to hello_world should be the "control network" IPs, or the IP addresses of the first NIC in your case. eRPC picks an available DPDK-capable port with a different DPDK-specific mechanism.

Getting an IP for the second NIC is tricky. On Azure, I'm assuming that the second NIC is called "eth1" as a temporary hack, and it has an assigned IP address.. Then eRPC gets the IP address of this NIC like so: https://github.com/erpc-io/eRPC/blob/master/src/transport_impl/dpdk/dpdk_transport.cc#L277. You'll probably need to modify this if your interface is named different than "eth1".

[gshanemiller]

Thank you for your time here. In return I will provide a detailed eRPC+AWS setup guide and benchmark results.

My understanding re: this issue then is,

• One provides the IP address(es) of the control NIC. Since the DPDK NIC is reserved for DPDK which does not implement a full IP stack, no IP address can be assigned to it anyway.
• The code through DPDK API likely can discover the PFs and VFs per PF on each machine

I'd appreciate one additional bit of information if possible. Once the DPDK NIC is bound to vfio-pci the DPDK Linux manual at https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio mentions other setup steps including:

Create the desired number of VF devices echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs

Do you also need to do this before running eRPC code? Or does the eRPC code create VF devices programmatically as needed? What is the upper bound on the number of VF devices? For example, maybe one of these approaches makes sense?

• VF device count is bound from above by the number or RX/TX queues
• VF device count is bound from above by the amount of on-NIC memory
• VF device count is bound from above by manufacturer. Each NIC has some max; read specs
• VF device count is like the number of ports on a UNIX: 1000s are available and what you need depends on software: how many concurrent connections are needed?

Finally: is a VF device duplex (sends and receives)? Or just RX or just TX only?

Thank you.

[anujkaliaiitd]

Thanks!

Similar to the instructions for running eRPC on Azure, I think there's no need to create VFs on the second NIC. eRPC "takes over" the second NIC from the OS kernel via DPDK. I'm not sure if it's even possible to create a VF, since the NIC exposed to the VM is already a VF, from the perspective of the physical host.

DPDK has some instructions specifically for ENA: https://doc.dpdk.org/guides/nics/ena.html. These don't seem to mention creating a VF. You might even try binding to igb_uio instead of vfio-pci.

Please LMK if I should clarify anything.

[gshanemiller]

Resolved - I will cycle back with AWS perf data & setup instructions ASAP.

[shenweihai1]

@gshanemiller do you mind posting the steps about how to run the eRPC experiment on the aws? I can't get it to work on aws, the eRPC complains Device RX ring too small which seems it can't find a second DPDK network interface (I also modified hardcoded eth1 to the correct one).