doonhammer commented 2 years ago

Thanks for getting the new GTP-U version out. I want to make sure I am using it correctly as I am seeing some behavior I do not understand.

My DUT is a firewall acting as a "dump in the wire" so it makes not changes to either the outer GTP-U tunnel or the inner flows. In my DUT logs I see the correct flows and I am not seeing any errors so it appears as if everything is working correctly. I am using Mellanox 100Gbps NICS on Centos 8. Running without GTP-U everything works as expected.

I set the server up with $ sudo ./t-rex-64 -i --astf --software --tso-disable -c 16

The client I go through the following steps:

$ ./trex-console

trex> tunnel --type gtpu trex> tunnels_topo load -f astf/gtpu_topo.py trex>start -f astf/http_simple.py -m 20000 -d 300

On the server stats output I see

-Per port stats table ports | 0 | 1

opackets | 4145979 | 1 obytes | 472641606 | 64 ipackets | 15 | 4100081 ibytes | 1124 | 483808899 ierrors | 0 | 0 oerrors | 0 | 0 Tx Bw | 253.63 Mbps | 0.00 bps

-Global stats enabled Cpu Utilization : 3.6 % 0.9 Gb/core Platform_factor : 1.0
Total-Tx : 253.63 Mbps
Total-Rx : 258.24 Mbps
Total-PPS : 278.11 Kpps
Total-CPS : 55.59 Kcps

Expected-PPS : 0.00 pps
Expected-CPS : 0.00 cps
Expected-L7-BPS : 0.00 bps

Active-flows : 295940 Clients : 0 Socket-util : 0.0000 %
Open-flows : 931694 Servers : 0 Socket : 0 Socket/Clients : -nan drop-rate : 0.00 bps
current time : 113.1 sec
test duration : 0.0 sec

One port 1 I do not see any Tx BW.

Similarly from tui I do not see any output from the TX bps Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 253.54 Mbps
version : ASTF @ v2.97 total_tx_L1 : 298.02 Mbps
cpu_util. : 4.66% @ 16 cores (16 per dual port) total_rx : 257.02 Mbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 278.01 Kpps
async_util. : 0% / 0.75 bps drop_rate : 0 bps
total_cps. : 55.63 Kcps queue_full : 0 pkts

Port Statistics

port	0	1	total -----------+-------------------+-------------------+------------------ owner
Tx bps L2	253.54 Mbps	0 bps	253.54 Mbps
Tx bps L1	298.02 Mbps	0 bps	298.02 Mbps
Tx pps	278.01 Kpps	0 pps	278.01 Kpps
Line Util.	0.3 %	0 %
---
Rx bps	▼▼▼ 10.13 bps	257.02 Mbps	257.02 Mbps
Rx pps	0.02 pps	272.27 Kpps	272.27 Kpps
----
opackets	34225734	0	34225734
ipackets	21	33611897	33611918
obytes	3901733676	0	3901733676
ibytes	1571	3966202939	3966204510
tx-pkts	34.23 Mpkts	0 pkts	34.23 Mpkts
rx-pkts	21 pkts	33.61 Mpkts	33.61 Mpkts
tx-bytes	3.9 GB	0 B	3.9 GB
rx-bytes	1.57 KB	3.97 GB	3.97 GB
-----
oerrors	0	0	0
ierrors	0	0	0

Also I would expect a higher throughput the % CPU is low on TREX and is also low on the DUT.

Regards

John

hhaim commented 2 years ago

@doonhammer with GTPU feature we expect that the DUT will terminate the GTPU session from client side and server side will be without tunnel. With loopback (your test) it does not happen. We have a feature to simulate "DUT" ( that remove the tunnel)

try this:

tunnel --type gtpu --loopback

doonhammer commented 2 years ago

Thanks guys helpful as always. Now I am seeing traffic in both directions with --loopback but I am seeing a very high drop rate on TREX, the DUT does not see drops, and it is not seeing heavy CPU load

trex>start -f astf/http_simple.py -m 5000 -d 300

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 2.81 Gbps
version : ASTF @ v2.97 total_tx_L1 : 2.87 Gbps
cpu_util. : 8.65% @ 16 cores (16 per dual port) total_rx : 228.51 Mbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 401.26 Kpps
async_util. : 0% / 25.47 bps drop_rate : 2.58 Gbps
total_cps. : 13.85 Kcps queue_full : 0 pkts

Port Statistics

port	0	1	total -----------+-------------------+-------------------+------------------ owner
Tx bps L2	184.81 Mbps	2.62 Gbps	2.81 Gbps
Tx bps L1	209.1 Mbps	2.66 Gbps	2.87 Gbps
Tx pps	151.82 Kpps	249.45 Kpps	401.26 Kpps
Line Util.	0.21 %	2.66 %
---
Rx bps	38.78 Mbps	189.73 Mbps	228.51 Mbps
Rx pps	41.55 Kpps	151.87 Kpps	193.41 Kpps
----
opackets	138058637	153117433	291176070
ipackets	27436741	117777832	145214573
obytes	19259941524	196381839750	215641781274
ibytes	3190929528	17309592410	20500521938
tx-pkts	138.06 Mpkts	153.12 Mpkts	291.18 Mpkts
rx-pkts	27.44 Mpkts	117.78 Mpkts	145.21 Mpkts
tx-bytes	19.26 GB	196.38 GB	215.64 GB
rx-bytes	3.19 GB	17.31 GB	20.5 GB
-----
oerrors	0	124,290,567	124,290,567
ierrors	0	0	0

status: /

Press 'ESC' for navigation panel... status:

tui>

Any ideas?

Regards

John

syaakov commented 2 years ago

@doonhammer Can you send us the CPU utilization per thread ? You can do that using tui in Trex console. In tui press Esc button and then press 'u'.

If you are noticing that one or more of the threads have very high CPU utilization, then you probably have bad distribution. You can solve this by adding more tunnels. There is more information about this here: https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_gtpu_traffic_tutorials under: Tutorial: Bad distribution with GTPU traffic section.

doonhammer commented 2 years ago

@syaakov thanks for the info, yes I was getting high CPU utilization all on one thread. I created four tunnels and it is a little better but I am still seeing a high drop rate. Here is my output and modified tunnel code - am I missing something here?

Regards

John

Example of Topology profile

from trex.astf.api import * from trex.astf.tunnels_topo import TunnelsTopo

def get_topo(): topo = TunnelsTopo()

topo.add_tunnel_ctx(
    src_start = '16.0.0.0',
    src_end  = '16.0.0.63',
    initial_teid = 0,
    teid_jump = 1,
    sport = 5000,
    version = 4,
    tunnel_type = 1,
    src_ip = '1.1.1.11',
    dst_ip = '12.2.2.2',
    activate = True
)

topo.add_tunnel_ctx(
    src_start = '16.0.0.64',
    src_end  = '16.0.0.127',
    initial_teid = 0,
    teid_jump = 1,
    sport = 6000,
    version = 4,
    tunnel_type = 1,
    src_ip = '1.1.2.11',
    dst_ip = '12.2.3.2',
    activate = True
)

topo.add_tunnel_ctx(
    src_start = '16.0.0.128',
    src_end  = '16.0.0.195',
    initial_teid = 0,
    teid_jump = 1,
    sport = 7000,
    version = 4,
    tunnel_type = 1,
    src_ip = '1.1.3.11',
    dst_ip = '12.2.4.2',
    activate = True
)

topo.add_tunnel_ctx(
    src_start = '16.0.0.196',
    src_end  = '16.0.0.255',
    initial_teid = 0,
    teid_jump = 1,
    sport = 8000,
    version = 4,
    tunnel_type = 1,
    src_ip = '1.1.4.11',
    dst_ip = '12.2.5.2',
    activate = True
)

return topo

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 5.17 Gbps ▲
version : ASTF @ v2.97 total_tx_L1 : 5.28 Gbps ▲
cpu_util. : 14.08% @ 16 cores (16 per dual port) total_rx : 363.05 Mbps ▲
rx_cpu_util. : 0.0% / 0 pps total_pps : 675.66 Kpps ▲
async_util. : 0% / 31.87 bps drop_rate : 4.81 Gbps
total_cps. : 26.7 Kcps ▲ queue_full : 0 pkts

Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14 -----------+-----+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---- 0 (0,1) | 10 | 10 | 9 | 10 | 10 | 9 | 9 | 10 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 1 (0,1) | 9 | 10 | 9 | 9 | 9 | 10 | 9 | 10 | 10 | 8 | 10 | 10 | 9 | 9 | 9 | 10 2 (0,1) | 10 | 11 | 10 | 10 | 9 | 10 | 10 | 10 | 11 | 9 | 10 | 10 | 10 | 9 | 9 | 10 3 (0,1) | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 10 | 10 | 9 | 10 | 10 | 10 | 9 | 9 | 9 4 (0,1) | 9 | 10 | 9 | 10 | 9 | 9 | 10 | 9 | 9 | 9 | 9 | 9 | 10 | 8 | 8 | 9 5 (0,1) | 43 | 47 | 45 | 46 | 38 | 37 | 28 | 22 | 33 | 19 | 25 | 17 | 19 | 20 | 18 | 17 6 (0,1) | 26 | 29 | 26 | 29 | 26 | 22 | 19 | 16 | 20 | 15 | 21 | 14 | 14 | 16 | 13 | 13 7 (0,1) | 9 | 9 | 9 | 10 | 10 | 9 | 9 | 11 | 10 | 8 | 9 | 9 | 9 | 9 | 9 | 9 8 (0,1) | 9 | 9 | 9 | 9 | 10 | 9 | 10 | 11 | 10 | 10 | 9 | 9 | 10 | 9 | 9 | 9 9 (0,1) | 10 | 10 | 10 | 10 | 9 | 10 | 9 | 11 | 10 | 9 | 10 | 10 | 10 | 9 | 9 | 10 10 (0,1) | 10 | 9 | 10 | 10 | 11 | 9 | 8 | 11 | 10 | 9 | 9 | 10 | 9 | 9 | 9 | 8 11 (0,1) | 10 | 10 | 10 | 10 | 10 | 10 | 8 | 9 | 9 | 9 | 9 | 8 | 9 | 9 | 9 | 9 12 (0,1) | 10 | 9 | 10 | 11 | 11 | 10 | 9 | 10 | 10 | 9 | 10 | 9 | 9 | 10 | 10 | 10 13 (0,1) | 26 | 29 | 26 | 28 | 26 | 21 | 16 | 17 | 23 | 13 | 19 | 13 | 14 | 17 | 14 | 13 14 (0,1) | 11 | 10 | 10 | 12 | 11 | 10 | 9 | 8 | 9 | 9 | 10 | 10 | 9 | 9 | 9 | 9 15 (0,1) | 13 | 12 | 14 | 13 | 13 | 14 | 11 | 12 | 12 | 11 | 12 | 11 | 12 | 12 | 12 | 12

Mbuf Util

       |   64b   |  128b   |  256b   |  512b   |  1024b  |  2048b  |  4096b  |   9kb   | RAM(MB)

-----------+---------+---------+---------+---------+---------+---------+---------+---------+-------- Total: | 81900 | 40950 | 16384 | 16384 | 16384 | 36855 | 16384 | 147456 | 1509 Used: | | | | | | | | |
Socket 0: | 161 | 437 | 1 | 0 | 0 | 16 | 0 | 130779 | 1177 Percent: | 0% | 1% | 0% | 0% | 0% | 0% | 0% | 88% |

status: -

browse: 'ESC' - console, 'q' - quit, 'd' - dashboard, 'u' - util, 't' - astf, 'l' - latency, ustats:

doonhammer commented 2 years ago

@syaakov I extend my tests to create 16 GTP-U tunnels (I have allocated 16 cores to TREX) and the load appears to be Archive.zip I created two files one for the GTP-U tunnels and one to send traffic. The resulting traffic appears to be evenly distributed but there is still a high drop rate.

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 11.77 Gbps
version : ASTF @ v2.97 total_tx_L1 : 12.19 Gbps
cpu_util. : 59.13% @ 16 cores (16 per dual port) total_rx : 2.27 Gbps ▲
rx_cpu_util. : 0.0% / 0 pps total_pps : 2.64 Mpps
async_util. : 0% / 9.24 bps drop_rate : 9.5 Gbps
total_cps. : 79.97 Kcps queue_full : 0 pkts

Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14 -----------+-----+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---- 0 (0,1) | 58 | 59 | 59 | 60 | 56 | 56 | 54 | 55 | 54 | 52 | 55 | 52 | 49 | 48 | 50 | 49 1 (0,1) | 58 | 59 | 60 | 58 | 58 | 56 | 58 | 55 | 53 | 51 | 51 | 54 | 48 | 47 | 50 | 46 2 (0,1) | 59 | 58 | 58 | 62 | 59 | 56 | 56 | 56 | 56 | 54 | 53 | 53 | 52 | 50 | 47 | 46 3 (0,1) | 58 | 58 | 59 | 61 | 57 | 56 | 55 | 54 | 55 | 53 | 52 | 56 | 51 | 49 | 48 | 46 4 (0,1) | 59 | 60 | 58 | 59 | 58 | 58 | 55 | 54 | 54 | 51 | 51 | 55 | 50 | 51 | 50 | 45 5 (0,1) | 61 | 62 | 63 | 62 | 58 | 58 | 57 | 58 | 58 | 54 | 55 | 59 | 54 | 54 | 49 | 50 6 (0,1) | 59 | 61 | 61 | 59 | 57 | 58 | 56 | 56 | 54 | 54 | 52 | 55 | 50 | 49 | 49 | 51 7 (0,1) | 58 | 59 | 60 | 58 | 58 | 56 | 55 | 57 | 57 | 56 | 55 | 56 | 51 | 47 | 48 | 46 8 (0,1) | 59 | 59 | 60 | 60 | 58 | 57 | 55 | 57 | 56 | 57 | 55 | 53 | 51 | 51 | 51 | 48 9 (0,1) | 58 | 60 | 59 | 60 | 56 | 57 | 55 | 55 | 56 | 53 | 53 | 54 | 50 | 51 | 48 | 46 10 (0,1) | 58 | 61 | 58 | 60 | 56 | 57 | 54 | 55 | 52 | 51 | 52 | 55 | 52 | 52 | 50 | 47 11 (0,1) | 59 | 60 | 60 | 59 | 57 | 58 | 55 | 52 | 53 | 52 | 51 | 53 | 49 | 49 | 47 | 49 12 (0,1) | 59 | 58 | 60 | 60 | 57 | 58 | 56 | 55 | 55 | 53 | 53 | 54 | 49 | 48 | 50 | 47 13 (0,1) | 58 | 60 | 58 | 59 | 57 | 56 | 56 | 56 | 54 | 52 | 53 | 54 | 49 | 51 | 51 | 50 14 (0,1) | 58 | 60 | 60 | 59 | 57 | 56 | 55 | 57 | 56 | 52 | 53 | 54 | 50 | 49 | 49 | 49 15 (0,1) | 73 | 75 | 75 | 73 | 70 | 70 | 67 | 69 | 69 | 69 | 70 | 68 | 62 | 60 | 61 | 62

Mbuf Util

       |   64b   |  128b   |  256b   |  512b   |  1024b  |  2048b  |  4096b  |   9kb   | RAM(MB)

-----------+---------+---------+---------+---------+---------+---------+---------+---------+-------- Total: | 81900 | 40950 | 16384 | 16384 | 16384 | 36855 | 16384 | 147456 | 1509 Used: | | | | | | | | |
Socket 0: | 454 | 971 | 1 | 0 | 0 | 16 | 0 | 130336 | 1173 Percent: | 0% | 2% | 0% | 0% | 0% | 0% | 0% | 88% |

status: |

Press 'ESC' for navigation panel... status:

Any other hints or suggestions?

Regards

John

syaakov commented 2 years ago

@doonhammer Can you send us the astf counters, and the dynamic counters that holds the GTPU counters ? for astf counters: In tui press Esc button and then press 't'.

for dynamic counters: In tui press Esc button and then 't' and then press 'm' button.

And by the way, Do you see any error when you start this traffic without GTPU ?

doonhammer commented 2 years ago

@syaakov I run have run the test without GTP-U enabled and I see no errors and about 8-9 Gbps throughput trex> start -f astf/http_multiple_16.py -m 2000 -d 600

When I run with gtp-u enabled trex> tunnel --type gtpu --loopback trex> tunnels_topo load -f astf/gtpu_topo_multiple_16.py trex>start -f astf/http_multiple.py -m 1000 -d 300

I see the following

Regards

John

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 6.53 Gbps
version : ASTF @ v2.97 total_tx_L1 : 6.68 Gbps
cpu_util. : 30.63% @ 16 cores (16 per dual port) total_rx : 578.79 Mbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 983.89 Kpps
async_util. : 0% / 5.46 bps drop_rate : 5.95 Gbps
total_cps. : 31.96 Kcps queue_full : 0 pkts

Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14 -----------+-----+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---- 0 (0,1) | 23 | 23 | 23 | 23 | 22 | 22 | 22 | 22 | 21 | 22 | 22 | 22 | 21 | 22 | 20 | 21 1 (0,1) | 22 | 23 | 22 | 21 | 22 | 21 | 22 | 22 | 22 | 22 | 23 | 23 | 22 | 21 | 21 | 21 2 (0,1) | 21 | 21 | 21 | 21 | 21 | 20 | 21 | 22 | 21 | 21 | 23 | 21 | 21 | 20 | 20 | 21 3 (0,1) | 23 | 23 | 23 | 23 | 22 | 23 | 22 | 22 | 22 | 22 | 22 | 23 | 22 | 22 | 21 | 20 4 (0,1) | 21 | 21 | 21 | 20 | 21 | 21 | 21 | 22 | 22 | 21 | 21 | 21 | 21 | 20 | 18 | 20 5 (0,1) | 23 | 23 | 23 | 23 | 22 | 24 | 24 | 24 | 24 | 23 | 23 | 22 | 22 | 22 | 21 | 22 6 (0,1) | 22 | 22 | 22 | 21 | 22 | 23 | 23 | 23 | 23 | 22 | 22 | 21 | 21 | 21 | 20 | 20 7 (0,1) | 23 | 23 | 23 | 23 | 22 | 22 | 22 | 22 | 21 | 22 | 22 | 22 | 22 | 22 | 20 | 20 8 (0,1) | 36 | 36 | 38 | 35 | 36 | 37 | 36 | 39 | 37 | 35 | 35 | 37 | 33 | 31 | 29 | 33 9 (0,1) | 37 | 35 | 38 | 38 | 37 | 39 | 38 | 37 | 38 | 36 | 34 | 34 | 36 | 30 | 29 | 30 10 (0,1) | 37 | 38 | 37 | 37 | 38 | 37 | 37 | 36 | 37 | 35 | 37 | 35 | 32 | 32 | 29 | 33 11 (0,1) | 37 | 39 | 38 | 38 | 34 | 37 | 35 | 36 | 37 | 36 | 34 | 32 | 32 | 31 | 26 | 31 12 (0,1) | 37 | 37 | 37 | 37 | 35 | 38 | 34 | 36 | 35 | 37 | 35 | 35 | 32 | 31 | 28 | 31 13 (0,1) | 37 | 39 | 38 | 37 | 37 | 35 | 39 | 39 | 35 | 35 | 35 | 35 | 36 | 32 | 30 | 32 14 (0,1) | 38 | 38 | 41 | 36 | 35 | 39 | 36 | 38 | 35 | 35 | 35 | 35 | 33 | 31 | 29 | 31 15 (0,1) | 46 | 46 | 49 | 47 | 46 | 44 | 46 | 48 | 45 | 44 | 47 | 46 | 40 | 39 | 36 | 38

Mbuf Util

       |   64b   |  128b   |  256b   |  512b   |  1024b  |  2048b  |  4096b  |   9kb   | RAM(MB)

-----------+---------+---------+---------+---------+---------+---------+---------+---------+-------- Total: | 81900 | 40950 | 16384 | 16384 | 16384 | 36855 | 16384 | 147456 | 1509 Used: | | | | | | | | |
Socket 0: | 355 | 746 | 1 | 0 | 0 | 16 | 0 | 130537 | 1174 Percent: | 0% | 1% | 0% | 0% | 0% | 0% | 0% | 88% |

status: \

browse: 'ESC' - console, 'q' - quit, 'd' - dashboard, 'u' - util, 't' - astf, 'l' - latency, ustats:

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 6.53 Gbps
version : ASTF @ v2.97 total_tx_L1 : 6.69 Gbps
cpu_util. : 29.71% @ 16 cores (16 per dual port) total_rx : 580.95 Mbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 987.16 Kpps
async_util. : 0% / 5.46 bps drop_rate : 5.95 Gbps
total_cps. : 32.01 Kcps queue_full : 0 pkts

Traffic stats summary.

                       |      client       |      server       |

---------------------------+-------------------+-------------------+--------------------------- m_active_flows	2096066	152780	active open flows m_est_flows	2096011	152780
-
TCP
-
tcps_connattempt	2522698	0	connections initiated
tcps_accepts	0	2522620	connections accepted
tcps_connects	2522643	2522620	connections established
tcps_closed	426632	2369840	conn. closed (includes drops)
tcps_segstimed	5045341	2522620	segs where we tried to get rtt
tcps_rttupdated	5028416	2522620	times we succeeded
tcps_sndtotal	22080265	59061225	total packets sent
tcps_sndpack	2522643	25226200	data packets sent
tcps_sndbyte	628151802	80960966280	data bytes sent by application
tcps_sndbyte_ok	628138107	36527537600	data bytes sent by tcp
tcps_sndctrl	2949331	16470904	control (SYN	FIN	RST) packets sent
tcps_sndacks	14102463	5029645	ack-only packets sent

status: /

browse: 'ESC' - console, 'q' - quit, 'd' - dashboard, 'u' - util, 't' - astf, 'l' - latency, astats: 'c' - clear, 'Up' - scroll up, 'Down' - scroll down, 'Left' - previous TG, 'Right' - next TG, 'm' - dynamic counters on/off,

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 6.54 Gbps
version : ASTF @ v2.97 total_tx_L1 : 6.7 Gbps
cpu_util. : 30.43% @ 16 cores (16 per dual port) total_rx : 583.33 Mbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 989.46 Kpps
async_util. : 0% / 5.46 bps drop_rate : 5.96 Gbps
total_cps. : 32.05 Kcps queue_full : 0 pkts

Traffic dynamic counter stats summary.

                    |      client       |      server       |

------------------------+-------------------+-------------------+------------------------

| | |
GTPU | | |
| | |
m_tunnel_encapsulated | 30842071 | 57484590 | num of encapsulated pkts m_tunnel_decapsulated | 9491406 | 30841120 | num of decapsulated pkts m_err_tunnel_drops | 6 | 20072719 | num of pkts drop
m_err_tunnel_no_context | 0 | 20072713 | num of pkts without ctx m_err_l3_hdr | 6 | 6 | num of pkts with wrong l3 hdr

status: /

browse: 'ESC' - console, 'q' - quit, 'd' - dashboard, 'u' - util, 't' - astf, 'l' - latency, astats: 'c' - clear, 'Up' - scroll up, 'Down' - scroll down, 'Left' - previous TG, 'Right' - next TG, 'm' - dynamic counters on/off,

syaakov commented 2 years ago

@doonhammer you haven't sent me all of the ASTF counters. In the ASTF counters window, you can use the arrows to scroll down and up, and there are more counters if you scroll down. You can also print the counters using the ASTF client instead of using the console.

It seems that you have a lot of rst packets because the average packet size is: m_avg_size | 16.7 B | 8.62 B | average pkt size. You can see the number of sent rst packets if you scroll down in the ASTF counters window.

Can you try this test with and without GTPU mode and send me all of the ASTF counters.

doonhammer commented 2 years ago

@syaakov appreciate all the help. Here are the stats with no GTP-U. I am seeing roughly the same throughput and packet count on the DUT as I am seeing on TREX.

trex>stats --za Traffic stats summary.

                       |      client       |      server       |

---------------------------+-------------------+-------------------+--------------------------- m_active_flows	22569	22490	active open flows m_est_flows	22496	22490
m_traffic_duration	0 sec	0 sec	measured traffic duration
-
TCP
-
tcps_connattempt	1670287	0	connections initiated
tcps_accepts	0	1670170	connections accepted
tcps_connects	1670214	1670170	connections established
tcps_closed	1647718	1647680	conn. closed (includes drops)
tcps_segstimed	5004339	6667071	segs where we tried to get rtt
tcps_rttupdated	5004135	8335257	times we succeeded
tcps_delack	3336062	0	delayed acks sent
tcps_sndtotal	10008710	41723396	total packets sent
tcps_sndpack	1670214	38370453	data packets sent
tcps_sndbyte	415901463	53602435980	data bytes sent by application
tcps_sndbyte_ok	415883286	53547093010	data bytes sent by tcp
tcps_sndctrl	1670287	0	control (SYN	FIN	RST) packets sent
tcps_sndacks	6667941	3335361	ack-only packets sent
tcps_rcvtotal	41706421	10008050	total packets received
tcps_rcvpack	40033967	3333963	packets received in sequence
tcps_rcvbyte	53546390044	415872330	bytes received in sequence
tcps_rcvackpack	5004135	8335257	rcvd ack packets
tcps_rcvackbyte	415867848	53446726526	tx bytes acked by rcvd acks
tcps_rcvackbyte_of	1663769	1663730	tx bytes acked by rcvd acks - overflow acked
tcps_rcvoffloads	0	0	receive offload packets by software
tcps_preddat	36698778	0	times hdr predict ok for data pkts
tcps_drops	0	0	connections dropped
tcps_conndrops	0	0	embryonic connections dropped
tcps_timeoutdrop	0	0	conn. dropped in rxmt timeout
tcps_rexmttimeo	268	1459	retransmit timeouts
tcps_rexmttimeo_syn	1014	235	retransmit SYN timeouts
tcps_persisttimeo	0	0	persist timeouts
tcps_keeptimeo	0	0	keepalive timeouts
tcps_keepprobe	0	0	keepalive probes sent
tcps_keepdrops	0	0	connections dropped in keepalive
tcps_testdrops	0	0	connections dropped by user at timeout (no-close flag --nc)
tcps_sndrexmitpack	268	17582	data packets retransmitted
tcps_sndrexmitbyte	66732	22647718	data bytes retransmitted
tcps_sndprobe	0	0	window probes sent
tcps_sndurg	0	0	packets sent with URG only
tcps_sndwinup	0	0	window update-only packets sent
tcps_rcvbadsum	0	0	packets received with ccksum errs
tcps_rcvbadoff	0	0	packets received with bad offset
tcps_rcvshort	0	0	packets received too short
tcps_rcvduppack	1	268	duplicate-only packets received
tcps_rcvdupbyte	0	66732	duplicate-only bytes received
tcps_rcvpartduppack	0	0	packets with some duplicate data
tcps_rcvpartdupbyte	0	0	dup. bytes in part-dup. packets
tcps_rcvoopackdrop	0	0	OOO packet drop due to queue len
tcps_rcvoobytesdrop	0	0	OOO bytes drop due to queue len
tcps_rcvoopack	3182	0	out-of-order packets received
tcps_rcvoobyte	4607536	0	out-of-order bytes received
tcps_rcvpackafterwin	0	0	packets with data after window
tcps_rcvbyteafterwin	0	0	bytes rcvd after window
tcps_rcvafterclose	0	0	packets rcvd after close
tcps_rcvwinprobe	0	0	rcvd window probe packets
tcps_rcvdupack	270	1977	rcvd duplicate acks
tcps_rcvacktoomuch	0	0	rcvd acks for unsent data
tcps_rcvwinupd	0	0	rcvd window update packets
tcps_pawsdrop	0	0	segments dropped due to PAWS
tcps_predack	0	3331437	times hdr predict ok for acks
tcps_persistdrop	0	0	timeout in persist state
tcps_badsyn	75	0	bogus SYN, e.g. premature ACK
tcps_reasalloc	251	0	allocate tcp reasembly ctx
tcps_reasfree	251	0	free tcp reasembly ctx
tcps_reas_hist_4	251	0	count of max queue <= 4
tcps_reas_hist_16	0	0	count of max queue <= 16
tcps_reas_hist_100	0	0	count of max queue <= 100
tcps_reas_hist_other	0	0	count of max queue > 100
tcps_nombuf	0	0	no mbuf for tcp - drop the packets
tcps_sack_recovery_episode	0	223	SACK recovery episodes
tcps_sack_rexmits	0	1257	SACK rexmit segments
tcps_sack_rexmit_bytes	0	1820136	SACK rexmit bytes
tcps_sack_rcv_blocks	268	2931	SACK blocks (options) received
tcps_sack_send_blocks	2931	268	SACK blocks (options) sent
tcps_sack_sboverflow	0	0	times scoreboard overflowed
-
UDP
-
udps_accepts	0	0	connections accepted
udps_connects	0	0	connections established
udps_closed	0	0	conn. closed (includes drops)
udps_sndbyte	0	0	data bytes transmitted
udps_sndpkt	0	0	data packets transmitted
udps_rcvbyte	0	0	data bytes received
udps_rcvpkt	0	0	data packets received
udps_keepdrops	0	0	keepalive drop
udps_nombuf	0	0	no mbuf
udps_pkt_toobig	0	0	packets transmitted too big
-
Flow Table
-
rss_redirect_rx	39099960	0	rss rx packets redirected
rss_redirect_tx	39099960	0	rss tx packets redirected
rss_redirect_drops	0	0	rss packets to redirect dropped
rss_redirect_queue_full	0	0	rss tx queue full
ignored_macs	0	0	ignored macs addr
ignored_ips	0	0	ignored ips addr
err_cwf	0	0	client pkt without flow
err_no_syn	0	0	server first flow packet with no SYN
err_len_err	0	0	pkt with length error
err_fragments_ipv4_drop	0	0	fragments_ipv4_drop
err_no_tcp_udp	0	0	no tcp/udp packet
err_no_template	0	0	server can't match L7 template
err_no_memory	0	0	No heap memory for allocating flows
err_dct	0	0	duplicate flow - more clients require
err_l3_cs	0	0	ip checksum error
err_l4_cs	0	0	tcp/udp checksum error
err_redirect_rx	0	0	redirect to rx error
redirect_rx_ok	0	0	redirect to rx OK
err_rx_throttled	0	0	rx thread was throttled
err_c_nf_throttled	0	0	client new flow throttled
err_c_tuple_err	0	0	client new flow, not enough clients
err_s_nf_throttled	0	0	server new flow throttled
err_flow_overflow	0	0	too many flows errors
defer_template	0	0	tcp L7 template matching deferred (by l7_map)
err_defer_no_template	0	0	server can't match L7 template (deferred by l7_map)

doonhammer commented 2 years ago

@syaakov running on GTP-U mode I get get the following:

CPU Stats trex>stats -c Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14 -----------+-----+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---- 0 (0,1) | 12 | 13 | 12 | 12 | 12 | 11 | 11 | 11 | 11 | 10 | 10 | 9 | 10 | 9 | 8 | 10 1 (0,1) | 13 | 14 | 13 | 13 | 12 | 12 | 10 | 9 | 9 | 8 | 8 | 8 | 8 | 7 | 7 | 9 2 (0,1) | 13 | 14 | 13 | 12 | 12 | 12 | 10 | 10 | 10 | 9 | 8 | 9 | 8 | 8 | 7 | 10 3 (0,1) | 13 | 14 | 13 | 13 | 12 | 11 | 10 | 9 | 9 | 9 | 9 | 8 | 8 | 8 | 7 | 8 4 (0,1) | 13 | 13 | 13 | 13 | 12 | 12 | 10 | 10 | 10 | 9 | 9 | 9 | 8 | 8 | 8 | 9 5 (0,1) | 12 | 13 | 12 | 12 | 12 | 13 | 11 | 11 | 10 | 10 | 9 | 10 | 9 | 8 | 8 | 10 6 (0,1) | 12 | 13 | 12 | 12 | 13 | 12 | 11 | 11 | 10 | 10 | 10 | 9 | 10 | 9 | 8 | 9 7 (0,1) | 12 | 13 | 13 | 12 | 12 | 12 | 10 | 10 | 9 | 10 | 9 | 8 | 9 | 9 | 8 | 9 8 (0,1) | 24 | 25 | 23 | 25 | 24 | 24 | 26 | 26 | 23 | 24 | 22 | 24 | 22 | 19 | 20 | 21 9 (0,1) | 25 | 25 | 25 | 25 | 24 | 25 | 26 | 23 | 22 | 23 | 22 | 21 | 21 | 20 | 20 | 21 10 (0,1) | 24 | 26 | 25 | 25 | 24 | 22 | 26 | 24 | 24 | 24 | 25 | 23 | 24 | 23 | 23 | 22 11 (0,1) | 24 | 25 | 25 | 24 | 24 | 24 | 22 | 23 | 21 | 23 | 22 | 21 | 21 | 20 | 20 | 19 12 (0,1) | 24 | 25 | 24 | 25 | 24 | 24 | 25 | 25 | 23 | 23 | 23 | 25 | 20 | 21 | 22 | 21 13 (0,1) | 25 | 25 | 24 | 26 | 23 | 26 | 23 | 22 | 23 | 23 | 23 | 20 | 22 | 20 | 19 | 18 14 (0,1) | 26 | 27 | 25 | 25 | 26 | 25 | 26 | 23 | 22 | 22 | 22 | 22 | 21 | 21 | 21 | 20 15 (0,1) | 31 | 33 | 33 | 31 | 30 | 28 | 28 | 28 | 29 | 26 | 26 | 27 | 26 | 25 | 25 | 25

Traffic Stats

Traffic stats summary.

                       |      client       |      server       |

---------------------------+-------------------+-------------------+--------------------------- m_active_flows	2094295	154072	active open flows m_est_flows	2094217	154072
m_traffic_duration	0 sec	0 sec	measured traffic duration
-
TCP
-
tcps_connattempt	3118620	0	connections initiated
tcps_accepts	0	3118520	connections accepted
tcps_connects	3118542	3118520	connections established
tcps_closed	1024325	2964448	conn. closed (includes drops)
tcps_segstimed	6237162	3118520	segs where we tried to get rtt
tcps_rttupdated	6220209	3118520	times we succeeded
tcps_delack	0	0	delayed acks sent
tcps_sndtotal	29563798	74883113	total packets sent
tcps_sndpack	3118542	31185200	data packets sent
tcps_sndbyte	776536380	100085780880	data bytes sent by application
tcps_sndbyte_ok	776516958	45156169600	data bytes sent by tcp
tcps_sndctrl	4142945	22154788	control (SYN	FIN	RST) packets sent
tcps_sndacks	19200560	6230010	ack-only packets sent
tcps_rcvtotal	9194361	9348530	total packets received
tcps_rcvpack	0	3118520	packets received in sequence
tcps_rcvbyte	0	776511480	bytes received in sequence
tcps_rcvackpack	6220209	3118520	rcvd ack packets
tcps_rcvackbyte	772315083	0	tx bytes acked by rcvd acks
tcps_rcvackbyte_of	0	0	tx bytes acked by rcvd acks - overflow acked
tcps_rcvoffloads	0	0	receive offload packets by software
tcps_preddat	0	0	times hdr predict ok for data pkts
tcps_drops	1024325	2964448	connections dropped
tcps_conndrops	0	0	embryonic connections dropped
tcps_timeoutdrop	0	2964449	conn. dropped in rxmt timeout
tcps_rexmttimeo	3101751	15313117	retransmit timeouts
tcps_rexmttimeo_syn	16	0	retransmit SYN timeouts
tcps_persisttimeo	0	0	persist timeouts
tcps_keeptimeo	17256179	4272	keepalive timeouts
tcps_keepprobe	16231854	4272	keepalive probes sent
tcps_keepdrops	1024325	0	connections dropped in keepalive
tcps_testdrops	0	0	connections dropped by user at timeout (no-close flag --nc)
tcps_sndrexmitpack	3101751	15313117	data packets retransmitted
tcps_sndrexmitbyte	772335999	22173393416	data bytes retransmitted
tcps_sndprobe	0	0	window probes sent
tcps_sndurg	0	0	packets sent with URG only
tcps_sndwinup	0	0	window update-only packets sent
tcps_rcvbadsum	0	0	packets received with ccksum errs
tcps_rcvbadoff	0	0	packets received with bad offset
tcps_rcvshort	0	0	packets received too short
tcps_rcvduppack	4272	3107160	duplicate-only packets received
tcps_rcvdupbyte	0	772322055	duplicate-only bytes received
tcps_rcvpartduppack	0	0	packets with some duplicate data
tcps_rcvpartdupbyte	0	0	dup. bytes in part-dup. packets
tcps_rcvoopackdrop	0	0	OOO packet drop due to queue len
tcps_rcvoobytesdrop	0	0	OOO bytes drop due to queue len
tcps_rcvoopack	0	0	out-of-order packets received
tcps_rcvoobyte	0	0	out-of-order bytes received
tcps_rcvpackafterwin	0	0	packets with data after window
tcps_rcvbyteafterwin	0	0	bytes rcvd after window
tcps_rcvafterclose	0	0	packets rcvd after close
tcps_rcvwinprobe	0	0	rcvd window probe packets
tcps_rcvdupack	9734	3111432	rcvd duplicate acks
tcps_rcvacktoomuch	0	0	rcvd acks for unsent data
tcps_rcvwinupd	0	0	rcvd window update packets
tcps_pawsdrop	0	0	segments dropped due to PAWS
tcps_predack	0	0	times hdr predict ok for acks
tcps_persistdrop	0	0	timeout in persist state
tcps_badsyn	0	0	bogus SYN, e.g. premature ACK
tcps_reasalloc	0	0	allocate tcp reasembly ctx
tcps_reasfree	0	0	free tcp reasembly ctx
tcps_reas_hist_4	0	0	count of max queue <= 4
tcps_reas_hist_16	0	0	count of max queue <= 16
tcps_reas_hist_100	0	0	count of max queue <= 100
tcps_reas_hist_other	0	0	count of max queue > 100
tcps_nombuf	0	0	no mbuf for tcp - drop the packets
tcps_sack_recovery_episode	0	0	SACK recovery episodes
tcps_sack_rexmits	0	0	SACK rexmit segments
tcps_sack_rexmit_bytes	0	0	SACK rexmit bytes
tcps_sack_rcv_blocks	3101667	0	SACK blocks (options) received
tcps_sack_send_blocks	0	3101695	SACK blocks (options) sent
tcps_sack_sboverflow	0	0	times scoreboard overflowed
-
UDP
-
udps_accepts	0	0	connections accepted
udps_connects	0	0	connections established
udps_closed	0	0	conn. closed (includes drops)
udps_sndbyte	0	0	data bytes transmitted
udps_sndpkt	0	0	data packets transmitted
udps_rcvbyte	0	0	data bytes received
udps_rcvpkt	0	0	data packets received
udps_keepdrops	0	0	keepalive drop
udps_nombuf	0	0	no mbuf
udps_pkt_toobig	0	0	packets transmitted too big
-
Flow Table
-
rss_redirect_rx	8619259	0	rss rx packets redirected
rss_redirect_tx	8619259	0	rss tx packets redirected
rss_redirect_drops	0	0	rss packets to redirect dropped
rss_redirect_queue_full	0	0	rss tx queue full
ignored_macs	0	0	ignored macs addr
ignored_ips	0	0	ignored ips addr
err_cwf	0	0	client pkt without flow
err_no_syn	0	20214610	server first flow packet with no SYN
err_len_err	0	0	pkt with length error
err_fragments_ipv4_drop	0	0	fragments_ipv4_drop
err_no_tcp_udp	0	0	no tcp/udp packet
err_no_template	0	0	server can't match L7 template
err_no_memory	0	0	No heap memory for allocating flows
err_dct	0	0	duplicate flow - more clients require
err_l3_cs	0	0	ip checksum error
err_l4_cs	0	0	tcp/udp checksum error
err_redirect_rx	0	0	redirect to rx error
redirect_rx_ok	0	0	redirect to rx OK
err_rx_throttled	0	2	rx thread was throttled
err_c_nf_throttled	96787	0	client new flow throttled
err_c_tuple_err	0	0	client new flow, not enough clients
err_s_nf_throttled	0	0	server new flow throttled
err_flow_overflow	13349	0	too many flows errors
defer_template	0	0	tcp L7 template matching deferred (by l7_map)
err_defer_no_template	0	0	server can't match L7 template (deferred by l7_map)

doonhammer commented 2 years ago

@syaakov and the Dynamic Statistics

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 0 bps
version : ASTF @ v2.97 total_tx_L1 : 0 bps
cpu_util. : 0.0% @ 16 cores (16 per dual port) total_rx : 176.32 bps ▼▼▼
rx_cpu_util. : 0.0% / 0 pps total_pps : 0 pps
async_util. : 0% / 0.82 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts

Traffic dynamic counter stats summary.

                    |      client       |      server       |

------------------------+-------------------+-------------------+------------------------

| | |
GTPU | | |
| | |
m_tunnel_encapsulated | 239728969 | 331838229 | num of encapsulated pkts m_tunnel_decapsulated | 55496753 | 239712478 | num of decapsulated pkts m_err_tunnel_drops | 54 | 165794500 | num of pkts drop
m_err_tunnel_no_context | 0 | 165794446 | num of pkts without ctx m_err_l3_hdr | 54 | 54 | num of pkts with wrong l3 hdr

syaakov commented 2 years ago

@doonhammer it seems that Trex receives packets that it doesn't generate. err_no_syn | 0 | 20214610 | server first flow packet with no SYN m_err_l3_hdr | 54 | 54 | num of pkts with wrong l3 hdr

Can you do this test:

run Trex server with 1 core: sudo ./t-rex-64 -i --astf --iom 0 -c 1 --software --tso-disable

in Trex console: trex>tunnel --type gtpu --loopback trex>tunnels_topo load -f astf/gtpu_topo.py trex>capture record start --tx 0 --rx 0 -l 100 trex>capture record start --tx 1 --rx 1 -l 100 trex>start -f astf/http_simple.py -m 0.1

after the capture is full save to cap files you can see if it's full by using the command: trex> capture

saving the to pcap files trex>capture record stop --id 1 -o your_path/client_side.pcap trex>capture record stop --id 2 -o your_path/server_side.pcap

These are the results you should expect. gtpu.zip

doonhammer commented 2 years ago

@syaakov thank you for all your help. I have the issue, it is the MTU/MSS. The DUT is set in default mode to 1500 MTU. For some reason the GTP-U traffic has a larger MTU causing drops. I modified the http_simple to set the MSS and also my other tests and everything works great. I set the MSS to 1424 and everything works.

So just not sure of the math used to create the MSS

Does this make sense?

Regards

John

syaakov commented 2 years ago

@doonhammer The encapsulation length with GTPU with ipv4 is: IPV4 header + UDP header + GTPU = 20 + 8 + 8 = 36 So: 1500 - GTPU_Encapsulation - IPv4 header - TCP header = 1500 - 36 - 20 - 20 = 1424

doonhammer commented 2 years ago

@syaakov Should something be added to the documentation, as the default MSS for non tunnel traffic needs to be changed in the scripts? Thank you for all your help tracking this down and implementing the GTP-U support most appreciated.

Regards

John

syaakov commented 2 years ago

@doonhammer I will address this in the GTPU documentation.

doonhammer commented 2 years ago

@syaakov Thanks again will close the issue.

cisco-system-traffic-generator / trex-core

GTP-U Statistics issue #837

-Per port stats table ports | 0 | 1

Example of Topology profile