search

emmericp / MoonGen

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.
MIT License
1.04k stars 234 forks source link

40G line rate with many-core machine #211

Open sekhonramneek opened 6 years ago

sekhonramneek commented 6 years ago

Hi I am trying to generate 40G Traffic using Moongen on a many-core machine. The max Tx rate that i can achieve is 35 Gbps for 1500 bytes pktsize, while its as low as 7Gbps Tx rate for 64 Bytes packet size. The machine acting as traffic generator is connected to another manycore machine running ovs kernel module which loops back packet to traffic generator. I am currently using the l3-load-latency.lua example for this.

When testing with 10G, full line rate was generated @14mpps, however, when testing with 40G NIC (Intel XL710), even 10G cannot be achieved using 64 Bytes packet size.

When looking at the output, there seems to be some issue with the Intel nic that i am using, Following is a snippet from the output: [INFO] Found 1 usable devices: Device 0: 3C:FD:FE:9E:D6:F8 (Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+) [WARN] Device 0 already configured, skipping initilization [INFO] Waiting for devices to come up... [INFO] Device 0 (3C:FD:FE:9E:D6:F8) is up: 40000 MBit/s [INFO] 1 device is up. [WARN] Per-queue rate limit is not supported on this device, setting per-device rate limit to 39987 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting). i40e_ethertype_filter_set(): Conflict with existing ethertype rules! [WARN] l2 filter error: Invalid argument [INFO] Performing ARP lookup on 10.0.1.100 [INFO] Performing ARP lookup on 10.0.1.100 [INFO] ARP lookup failed, using default destination mac address [INFO] ARP lookup failed, using default destination mac address i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 2.93 Mpps, 35205 Mbit/s (35673 Mbit/s with framing) [Device: id=0] RX: 2.33 Mpps, 28011 Mbit/s (28383 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 2.86 Mpps, 34445 Mbit/s (34903 Mbit/s with framing) [Device: id=0] RX: 2.38 Mpps, 28597 Mbit/s (28977 Mbit/s with framing) [Device: id=0] TX: 2.93 Mpps, 35221 Mbit/s (35690 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. C[Device: id=0] TX: 2.92 (StdDev 0.02) Mpps, 35110 (StdDev 293) Mbit/s (35577 Mbit/s with framing), total 23639650 packets with 35553990400 bytes (incl. CRC) [Device: id=0] RX: 2.38 (StdDev 0.00) Mpps, 28617 (StdDev 14) Mbit/s (28998 Mbit/s with framing), total 19212972 packets with 28896266688 bytes (incl. CRC) Samples: 1116, Average: 1705772.5 ns, StdDev: 2128344.4 ns, Quartiles: 48423.0/92603.0/4440258.0 ns [INFO] Saving histogram to 'histogram.csv'

Any help in this regard will be appreciated.

emmericp commented 6 years ago

Which firmware version are you using? It should be printed on startup during DPDK initialization.

Am 07.03.2018 um 02:11 schrieb sekhonramneek notifications@github.com:

Hi I am trying to generate 40G Traffic using Moongen on a many-core machine. The max Tx rate that i can achieve is 35 Gbps for 1500 bytes pktsize, while its as low as 7Gbps Tx rate for 64 Bytes packet size. The machine acting as traffic generator is connected to another manycore machine running ovs kernel module which loops back packet to traffic generator. I am currently using the l3-load-latency.lua example for this.

When testing with 10G, full line rate was generated @14mpps, however, when testing with 40G NIC (Intel XL710), even 10G cannot be achieved using 64 Bytes packet size.

When looking at the output, there seems to be some issue with the Intel nic that i am using, Following is a snippet from the output: [INFO] Found 1 usable devices: Device 0: 3C:FD:FE:9E:D6:F8 (Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+) [WARN] Device 0 already configured, skipping initilization [INFO] Waiting for devices to come up... [INFO] Device 0 (3C:FD:FE:9E:D6:F8) is up: 40000 MBit/s [INFO] 1 device is up. [WARN] Per-queue rate limit is not supported on this device, setting per-device rate limit to 39987 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting). i40e_ethertype_filter_set(): Conflict with existing ethertype rules! [WARN] l2 filter error: Invalid argument [INFO] Performing ARP lookup on 10.0.1.100 [INFO] Performing ARP lookup on 10.0.1.100 [INFO] ARP lookup failed, using default destination mac address [INFO] ARP lookup failed, using default destination mac address i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 2.93 Mpps, 35205 Mbit/s (35673 Mbit/s with framing) [Device: id=0] RX: 2.33 Mpps, 28011 Mbit/s (28383 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 2.86 Mpps, 34445 Mbit/s (34903 Mbit/s with framing) [Device: id=0] RX: 2.38 Mpps, 28597 Mbit/s (28977 Mbit/s with framing) [Device: id=0] TX: 2.93 Mpps, 35221 Mbit/s (35690 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. C[Device: id=0] TX: 2.92 (StdDev 0.02) Mpps, 35110 (StdDev 293) Mbit/s (35577 Mbit/s with framing), total 23639650 packets with 35553990400 bytes (incl. CRC) [Device: id=0] RX: 2.38 (StdDev 0.00) Mpps, 28617 (StdDev 14) Mbit/s (28998 Mbit/s with framing), total 19212972 packets with 28896266688 bytes (incl. CRC) Samples: 1116, Average: 1705772.5 ns, StdDev: 2128344.4 ns, Quartiles: 48423.0/92603.0/4440258.0 ns [INFO] Saving histogram to 'histogram.csv'

Any help in this regard will be appreciated.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

sekhonramneek commented 6 years ago

Hi Following is the output from the moongen startup: sudo ./build/MoonGen examples/l3-load-latency.lua 0 0 [WARN] malloc() allocates objects >= 1 MiB from LuaJIT memory space. [WARN] Install libjemalloc if you encounter out of memory errors. [INFO] Initializing DPDK. This will take a few seconds... EAL: Detected 128 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: Device is blacklisted, not initializing EAL: PCI device 0000:01:00.2 on NUMA socket 0 EAL: Device is blacklisted, not initializing EAL: PCI device 0000:01:00.3 on NUMA socket 0 EAL: Device is blacklisted, not initializing EAL: PCI device 0000:55:00.0 on NUMA socket 2 EAL: probe driver: 8086:1583 net_i40e EAL: PCI device 0000:55:00.1 on NUMA socket 2 EAL: Device is blacklisted, not initializing [INFO] Found 1 usable devices: Device 0: 3C:FD:FE:9E:D6:F8 (Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+) [WARN] Device 0 already configured, skipping initilization [INFO] Waiting for devices to come up... [INFO] Device 0 (3C:FD:FE:9E:D6:F8) is up: 40000 MBit/s [INFO] 1 device is up. [WARN] Per-queue rate limit is not supported on this device, setting per-device rate limit to 39987 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting). i40e_ethertype_filter_set(): Conflict with existing ethertype rules! [WARN] l2 filter error: Invalid argument [INFO] Performing ARP lookup on 10.0.1.100 [INFO] Performing ARP lookup on 10.0.1.100 [INFO] ARP lookup failed, using default destination mac address [INFO] ARP lookup failed, using default destination mac address i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 2.99 Mpps, 35971 Mbit/s (36449 Mbit/s with framing) [Device: id=0] RX: 1.96 Mpps, 23577 Mbit/s (23890 Mbit/s with framing) [Device: id=0] TX: 2.79 Mpps, 33525 Mbit/s (33971 Mbit/s with framing) [Device: id=0] RX: 2.07 Mpps, 24965 Mbit/s (25297 Mbit/s with framing) [Device: id=0] TX: 3.00 Mpps, 36051 Mbit/s (36530 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] RX: 2.07 Mpps, 24924 Mbit/s (25255 Mbit/s with framing) [Device: id=0] TX: 2.99 Mpps, 36026 Mbit/s (36505 Mbit/s with framing) [Device: id=0] RX: 2.08 Mpps, 25050 Mbit/s (25383 Mbit/s with framing) [Device: id=0] TX: 2.99 Mpps, 35980 Mbit/s (36458 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] RX: 2.11 Mpps, 25345 Mbit/s (25683 Mbit/s with framing) [Device: id=0] TX: 2.99 Mpps, 36015 Mbit/s (36494 Mbit/s with framing) [Device: id=0] RX: 2.08 Mpps, 25073 Mbit/s (25406 Mbit/s with framing) [Device: id=0] TX: 3.00 Mpps, 36041 Mbit/s (36520 Mbit/s with framing) [Device: id=0] RX: 2.07 Mpps, 24913 Mbit/s (25244 Mbit/s with framing) i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. i40e_check_fdir_programming_status(): unknown programming status reported, len = 60, id = 0. [Device: id=0] TX: 3.00 Mpps, 36051 Mbit/s (36530 Mbit/s with framing) [Device: id=0] RX: 2.07 Mpps, 24883 Mbit/s (25214 Mbit/s with framing) ^C[Device: id=0] TX: 2.96 (StdDev 0.08) Mpps, 35670 (StdDev 946) Mbit/s (36144 Mbit/s with framing), total 24732363 packets with 37197430752 bytes (incl. CRC) [Device: id=0] RX: 2.08 (StdDev 0.01) Mpps, 25022 (StdDev 159) Mbit/s (25355 Mbit/s with framing), total 17200798 packets with 25869952741 bytes (incl. CRC) Samples: 636, Average: 3559051.8 ns, StdDev: 2994590.9 ns, Quartiles: 74557.0/6021753.0/6156655.0 ns [INFO] Saving histogram to 'histogram.csv'

In addition, the info about the nic firmware and driver etc is as follows: sudo ethtool -i ens4f0 driver: i40e version: 1.4.25-k firmware-version: 6.01 0x800034a4 1.1747.0 expansion-rom-version: bus-info: 0000:55:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes

Thank you for your quick response. Any help in this regard is appreciated.

Thanks and Regards Rem

emmericp commented 6 years ago

We are currently also tracking down problems with firmware 6.01. These problems don't appear on firmware 5.04.

5.05 is also unfortunately broken in different ways (no timestamping, see errata sheet).

AqsaKashaf commented 3 years ago

Hi, Is there any update with this? I am not able to do hardware rate control on XL710 NICs.