sdnfv / openNetVM

A high performance container-based NFV platform from GW and UCR.
http://sdnfv.github.io/onvm/
Other
261 stars 134 forks source link

Can't make Pktgen and ONVM work on the same machine #259

Closed BuzzRage closed 3 years ago

BuzzRage commented 3 years ago

Bug Report

Current Behavior Hi, I try to make PktGen works with OpenNetVM on the same machine. This machine has two NIC ports. I would like to generate packet with PktGen on one port, and then forward packets to a distant machine by passing through OpenNetVM and using the second NIC port. Here is dpdk-devbind output:

./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:41:00.0 'Ethernet 10G 2P X520 Adapter 154d' drv=igb_uio unused=ixgbe
0000:41:00.1 'Ethernet 10G 2P X520 Adapter 154d' drv=igb_uio unused=ixgbe

Network devices using kernel driver
===================================
0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno1 drv=tg3 unused=igb_uio 
0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno2 drv=tg3 unused=igb_uio *Active*

I'm trying different ways but nothing seems to work. I think I am misunderstanding something.

Expected behavior/code We expect to have a packet generator that sends data to OpenNetVM.

Steps to reproduce I followed the wiki here and I ran the manager with ./go.sh 0,1,2 3 0xF8 -s stdout -v, I set the correct MAC address in /tools/Pktgen/OpenNetVM-Scripts/pktgen-config.lua and I ran ./run-pktgen.sh 1 but it gives the following error:

Copyright (c) <2010-2019>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 12 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Error - exiting with code: 1
  Cause: Cannot create lock on '/var/run/dpdk/rte/config'. Is another primary process running?
Pktgen done

So I added --proc-type=secondary for EAL arguments in this line of ./run-pktgen.sh: (cd "$PKTGEN_HOME" && sudo "$PKTGEN_BUILD" -c 0xff -n 3 "$BLACK_LIST" --proc-type=secondary -- -p 0x1 "$PORT_MASK" -P -m "[1:2].0" -f "$PKTGEN_CONFIG") and it gaves me another error:

Copyright (c) <2010-2019>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 12 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_3522_2d407878e324d4
EAL: Probing VFIO support...
EAL: Cannot initialize tailq: RTE_LUADATA
Tailq 0: qname:<RTE_LPM>, tqh_first:(nil), tqh_last:0x7efffffd27c
Tailq 1: qname:<RTE_LPM6>, tqh_first:(nil), tqh_last:0x7efffffd2ac
Tailq 2: qname:<RTE_ACL>, tqh_first:(nil), tqh_last:0x7efffffd2dc
Tailq 3: qname:<RTE_HASH>, tqh_first:0x7f00032ae80, tqh_last:0x7f000c37700
Tailq 4: qname:<RTE_FBK_HASH>, tqh_first:(nil), tqh_last:0x7efffffd33c
Tailq 5: qname:<RTE_MEMBER>, tqh_first:(nil), tqh_last:0x7efffffd36c
Tailq 6: qname:<RTE_EVENT_RING>, tqh_first:(nil), tqh_last:0x7efffffd39c
Tailq 7: qname:<RTE_MEMPOOL>, tqh_first:0x7f0002163c0, tqh_last:0x7f000835dc0
Tailq 8: qname:<RTE_RING>, tqh_first:0x7f0003ab0c0, tqh_last:0x7f000c3b940
Tailq 9: qname:<RTE_REORDER>, tqh_first:(nil), tqh_last:0x7efffffd42c
Tailq 10: qname:<RTE_KNI>, tqh_first:(nil), tqh_last:0x7efffffd45c
Tailq 11: qname:<VFIO_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x7efffffd48c
Tailq 12: qname:<UIO_RESOURCE_LIST>, tqh_first:0x7f0003e8e40, tqh_last:0x7f0002a7dc0
Tailq 13: qname:<VMBUS_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x7efffffd4ec
Tailq 14: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 15: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 16: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 17: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 18: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 19: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 20: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 21: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 22: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 23: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 24: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 25: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 26: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 27: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 28: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 29: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 30: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 31: qname:<>, tqh_first:(nil), tqh_last:(nil)
EAL: FATAL: Cannot init tail queues for objects
EAL: Cannot init tail queues for objects
Pktgen done

I also tried to go directly with this command sudo -E ./app/x86_64-native-linuxapp-gcc/pktgen -l 3-5 -n 1 --proc-type=secondary -- -P -m "[4:5].0" in pktgen-dpdk directory but I can't manage to having both PktGen and ONVM work.

When I only run ./run-pktgen.sh 1 without OpenNetVM's manager it gives me that:

Starting pktgen

Copyright (c) <2010-2019>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 12 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:41:00.0 on NUMA socket 1
EAL:   probe driver: 8086:154d net_ixgbe
EAL: PCI device 0000:41:00.1 on NUMA socket 1
EAL:   probe driver: 8086:154d net_ixgbe
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio

*** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<

Initialize Port 0 -- TxQ 1, RxQ 1,  Src MAC a0:36:9f:61:f0:00

Port  0: Link Down <Enable promiscuous mode>

WARNING: Nothing to do on lcore 3: exiting
WARNING: Nothing to do on lcore 4: exiting
WARNING: Nothing to do on lcore 5: exiting
WARNING: Nothing to do on lcore 6: exiting
WARNING: Nothing to do on lcore 7: exiting
  RX processing lcore:   1 rx:  1 tx:  0
  TX processing lcore:   2 rx:  0 tx:  1
PANIC in pktgen_main_tx_loop():
*** port 0 socket ID 1 has different socket ID for lcore 2 socket ID 0
7: [/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7ffff6cf0a3f]]
6: [/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ffff6fc76db]]
5: [./app/x86_64-native-linuxapp-gcc/pktgen(eal_thread_loop+0x1f1) [0x555555726071]]
4: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_launch_one_lcore+0x9f) [0x555555622f2f]]
3: [./app/x86_64-native-linuxapp-gcc/pktgen(+0xc9d9e) [0x55555561dd9e]]
2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc5) [0x5555555e4df4]]
1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2e) [0x55555572cf4e]]
Aborted
Pktgen done

Environment

Possible Solution I know that it is possible to have multiple DPDK application on the same machine. So it might be possible to have PktGen and ONVM on the same machine. I think the problem come from a misunderstanding of EAL configuration (especially core mapping).

twood02 commented 3 years ago

This is not something that we have tested so I'm not sure if it will work. In particular, if you want packets to leave pktgen and go to ONVM that might be tricky since pktgen tries to send directly out the NIC -- I think the packets would have to leave the system, go to a switch, and then be routed back to ONVM (or are you connecting one port directly to the other with a cable?)

In any case, it seems like there is some issue with your pktgen configuration if it will not run even without ONVM. We are preparing an updated release which will include newer versions of both DPDK and pktgen -- if you want to try this out now you can use the develop branch in this repository. That might help resolve your basic pktgen problems.

To run them together you will need to modify both the pktgen and onvm manager startup script. There are some DPDK flags you need to modify to restrict the process to only allocate a portion of the huge page memory and you need to use the --file-prefix option so they use different hugetablefs entries. See https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html

I would suggest trying to get pktgen plus a simple DPDK sample application to run (l2fwd). If that works then move on to pktgen + ONVM. I can't think of any reason why ONVM would prevent this setup (although as I mentioned above, I don't think there is an easy way to have the pktgen packets to go ONVM without leaving the machine and coming back).

As a final note, if your goal is simply to generate packets and send them to OVNM you might consider the speed_tester example using the option to load a PCAP. We also have a load_generator sample which is more configurable, although I think it had some bugs (maybe we resolved these, I forget).

BuzzRage commented 3 years ago

Thank you for your answer.

I will try to connect one port directly to the other, even if I guess the best way to send packets with pktgen to ONVM is to have 2 machines.

I am able to run pktgen without ONVM, but yes, I still need to read the documentation to understand DPDK configurations. I will test the develop branch, thank you for your advice !

I will also try to adapt the pktgen and onvm manager startup script to my environment with two different --file-prefix. Also I realized that I didn't disabled hyperthreading, maybe it's part of the problem.

I already used the speed_tester NF with .pcap files, but I'm not sure if packets are simply stored in memory before being played or if they are actually played on a NIC port using the network stack.

I will try your suggestions and post results here, thank you for your time.

twood02 commented 3 years ago

Please re-open this if you are still having problems.