mtcp-stack / mtcp

mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems
Other
1.98k stars 435 forks source link

How to bring Amazon-ENA-NIC up? #279

Open zarzen opened 4 years ago

zarzen commented 4 years ago

After bind my ens4 (which is AWS-ENA nic) to vfio-pci driver, the system cannot see the device anymore. I can successfully complete the compiling process. But don't know how to run epserver through that ens4 interface.

Network devices using DPDK-compatible driver
============================================
0000:00:04.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:03.0 'Elastic Network Adapter (ENA) ec20' if=ens3 drv=ena unused=vfio-pci *Active*
vincentmli commented 4 years ago

I am wondering why ENA NIC is bound to vfio-pci, do you get:

Set RTE_SDK env variable as Set RTE_TARGET env variable as x86_64-native-linuxapp-gcc Are you using an Intel NIC (y/n)?

after typing 35 Exit Script ?

what is command 'ip link list' look like? no dpdk0 interface ?

zarzen commented 4 years ago

Hi @vincentmli Another driver option is not available on EC2, thus, vfio-pci is used. (I think the problem is due to the secure boot option.)

If I choose y for "Are you using an Intel NIC (y/n)", then yes, there is dpdk0 interface. While I am not sure it is the correct choice. Because AWS-ENA clearly is not an Intel NIC.

ajamshed commented 4 years ago

Hi @zarzen,

I apologize for the delayed response. I have not really used AWS ENA NIC before. But I don't think it will be that difficult to set up the NIC for mTCP. Can you please first verify whether you are able to run sample dpdk application with the you setup? You can use any program in dpdk/examples for your tests. Please make sure that you have read the details mentioned in https://doc.dpdk.org/guides/nics/ena.html first.

Please let me know once you have successfully run simple dpdk example applications. I can then guide you into running mTCP applications thereafter.

vincentmli commented 4 years ago

If I choose y for "Are you using an Intel NIC (y/n)", then yes, there is dpdk0 interface. While I am not sure it is the correct choice. Because AWS-ENA clearly is not an Intel NIC.

while I run mTCP in VMware ESXi VM, the driver used in DPDK is VMware VMXNET3 PMD driver which I think is not Intel NIC, but I choosed the Intel NIC anyway and it works fine, though I have to change the TX ring size to 512 to match VMXNET3 PMD and disable hardware checksum when run configure. you may also need to enable debug CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=y in dpdk/x86_64-native-linuxapp-gcc/.config, recompile the DPDK, reload DPDK driver, rebind the NIC and recompile mTCP example app to see if you can run mTCP example app and if not, any debug error gives you any clue