Unable to launch application: Cause: No Ethernet ports - bye

[zkl94]

Hi Folks,

We recently developed a simple key-value server using Seastar. We used default setting to run the server during development (no native stack, no DPDK). In order to benchmark it, we tried to run it with native stack and DPDK support on an Ubuntu VM on Google Cloud Platform.

The VM is of type n1-standard-8 with 8 vCPUs and 30GB memory. It is equipped with two network interfaces: one for remote login (first NIC), one for enabling DPDK (second NIC).

$ lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:03.0 Non-VGA unclassified device: Red Hat, Inc Virtio SCSI
00:04.0 Ethernet controller: Red Hat, Inc Virtio network device
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device

First, we clone the seastar project, do everything necessary to build it with DPDK enabled. Then we compile our project. After compilation, our application can run smoothly (./a.out, without parameters).

To enable DPDK + native stack, we did the following:

1. shutdown the second NIC with ifconfig SECOND_NIC down
2. use the $SEASTAR/dpdk/usertools/dpdk-setup.sh to:
   • build the x86_64-native-linuxapp-gcc target
   • insert igb_uio module to the kernel
   • bind the second NIC to igb_uio module
   • Setup hugepage mappings for non-NUMA systems (with hints from $SEASTAR/dpdk/usertools/cpu_layout.py)
3. re-run the application with ./a.out --network-stack native --dpdk-pmd --dhcp 1 --smp 8 -m 2G and we get the following error:

EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye

NIC binding output from dpdk-setup.sh:

Network devices using DPDK-compatible driver
============================================
0000:00:05.0 'Virtio network device' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:00:04.0 'Virtio network device' if=ens4 drv=virtio-pci unused=igb_uio *Active*

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>

So even though there is a device with DPDK-compatible driver, seastar can't recognize it.

Could you give us any help?

[avikivity]

Perhaps it is the problem with hugepages, try to set them up.

[zkl94]

Thanks for the help. I really appreciate it!

It turns out that I didn't config hugepages in the right way. After specifying 1G hugepage size in kernel cmd parameter and mount it with hugetlbfs with fstab, the hugepage warning disappear. However, the 'No Ethernet ports' error still exists.

EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye

My NIC binding is still the same as above, where one device has DPDK-compatible driver.

I am now able to run ./a.out, ./a.out --dpdk-pmd, and ./a.out --network-stack native(with help from this tutorial) successfully. However, ./a.out --dpdk-pmd --network-stack native still shows the error above.

Could you provide more suggestions?

[avikivity]

Do the sample applications from dpdk work?

[vkrivopalov]

Hi Zhikun,

Can you try to run your application through the scripts/run_with_dpdk.sh helper script? It should bind the specified NIC for you to use it with DPDK.

Another thing to try is modprobe uio_pci_generic which enables the generic driver suitable for most DPDK-comparible NICs.

[luzhang92]

Have you solved this problem? I also have this problem and I am suspecting that my NICs don't support DPDK. Have you considered you NICs?

[shanshanpt]

I encounter the same problem as @zkl94. EAL: Detected 64 lcore(s) EAL: Probing VFIO support... EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye

And, how do you run ./a.out --network-stack native successful ? Are you configure a virbr0 ? If not, it will output an error "[shard 0] seastar - Exceptional future ignored: std::system_error (error system:2, No such file or directory), backtrace: 0x41a9e4"

[sassyn]

Same here at my side.

Maybe someone can help?

[talsaiag]

In my case (same error) turns out that DPDK requires a couple of libraries to be linked even if not used by the code (specifically the PWDs that are registered automatically on library load - which causes rte_eth_dev_count to return 0) Use the -Wl,--whole-archive,-ldpdk,--no-whole-archive during the linking stage to link with the whole dpdk library to fix the issue. I don't know the best way to add this to the seastar library itself (in the CMake) but this, for example, fixes the issue specifically for the apps folder:

diff --git a/apps/CMakeLists.txt b/apps/CMakeLists.txt
index d1a97279..c2b3188e 100644
--- a/apps/CMakeLists.txt
+++ b/apps/CMakeLists.txt
@@ -40,7 +40,7 @@ macro (seastar_add_app name)
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})

   target_link_libraries (${target}
-    PRIVATE seastar_private)
+    PRIVATE seastar_private -Wl,--whole-archive,-ldpdk,--no-whole-archive)

   set_target_properties (${target}
     PROPERTIES

@avikivity maybe you can help with incorporating this into the seastar library build process