T-rex 3.04 with Mellanox ConnectX-6 and dummy port shows errors instead of stats while sending traffic

[norg]

While I have some T-rex setups running with Intel XL710 to achieve 40G with one port I struggle with a 100G setup based on Mellanox ConnectX-6 cards.

So far OFED installation worked, T-Rex setup as well and I can even see the traffic on the DUT but the report from T-rex shows errors instead of the rate, which I was used to from the setup with Intel NICS and DPDK.

T-Rex output:

Per port stats table
      ports |               0
 -----------------------------------------------------------------------------------------
   opackets |               0
     obytes |               0
   ipackets |               0
     ibytes |               0
    ierrors |       446702141
    oerrors |               0
      Tx Bw |       0.00  bps

-Global stats enabled
 Cpu Utilization : 3.2  %  0.0 Gb/core
 Platform_factor : 1.0
 Total-Tx        :       0.00  bps
 Total-Rx        :       0.00  bps
 Total-PPS       :       0.00  pps
 Total-CPS       :      27.00 Kcps

 Expected-PPS    :       1.20 Mpps
 Expected-CPS    :      18.04 Kcps
 Expected-BPS    :       4.24 Gbps

 Active-flows    :    15439  Clients :      255   Socket-util : 0.0961 %
 Open-flows      :   107208  Servers :    65535   Socket :    15439 Socket/Clients :  60.5
 drop-rate       :       0.00  bps
 current time    : 5.2 sec
 test duration   : 3594.8 sec

Runcommand:

/t-rex-64 -f cap2/sfr.yaml -p -c 4 -m 8 --cfg /etc/trex_cfg.yaml

I use -p to have the full flow received by the single port of the DUT (it's the Suricata IDS so I just need passive traffic forwarding towards it).

The /etc/trex_cfg.yaml looks like this:

### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['82:00.1', 'dummy']
  port_info:
      - dest_mac: 94:6d:ae:c2:8d:11
        src_mac:  94:6d:ae:ef:b0:f9
      - dest_mac: 94:6d:ae:c2:8d:10
        src_mac:  94:6d:ae:ef:b0:f8

  platform:
      master_thread_id: 0
      latency_thread_id: 2
      dual_if:
        - socket: 1
          threads: [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63]

While 82:00.1 is the port of the Mellanox NIC on the machine doing the traffic replay and the first destination mac being the DUT. But even if I remove the port info section it still shows those errors.

This type of setup with the -p flag and also the dummy in the config works with the Intel setup where I see the traffic rate and all those stats (of course just for opackets and obytes while ipackets is 0).

Is this just an output issue? I would like to see the stats and especially traffic rate being sent so I can verify on the DUT side if the traffic rate is received.

Thanks

[norg]

Funfact, I tried v2.87 and there the stats look correct. So I wanted to try to find out which version introduced the change but starting with v2.88 I always run into this error:

EAL: /lib/x86_64-linux-gnu/libmlx5.so.1: version `MLX5_OFED' not found (required by so/x86_64/libmlx5-64.so)
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins

Seems to be DPDK Update related, so the hunt between v2.87 and v3.04 is just a bit too much :/

[norg]

v3.03 looks good, might be the same issue as seen in https://github.com/cisco-system-traffic-generator/trex-core/issues/1091

[Civil]

I've had the same problem as an author. With ConnectX-6, it works in a different way than above.

However, it works fine with ConnectX-5 and ConnectX-4 (I hope that will help to narrow it down).

I've tried git bisect'ing between v3.03 and v3.04, and indeed https://github.com/cisco-system-traffic-generator/trex-core/commit/b0bc6c731019cd7e32542b83a1900d1fe6810968 is the commit that broke it.

[hhaim]

@Civil thanks for bisect. the commit you mentioned is the DPDK upgrade, the but might be related to the new DPDK and that was the main objective of the new version

[Civil]

@hhaim if that helps, I've tried to adapt DPDK 23.11 in my fork: https://github.com/Civil/trex-core/tree/dpdk2311

But as I'm not that familiar with your build system and assumptions I probably did that a bit suboptimal.

It builds like that, it runs, however problem with ConnectX-6 persists:

Stats are broken, but now in a bit different way than before.

[Civil]

Overall, while porting, I had few small complications:

1. few functions changed their names, e.x. rte_thread_set_name(rte_thread_self(), "TRex RX");, also same for bonding (rte_eth_bond_slaves_get -> rte_eth_bond_members_get - and probably that would require renaming variables around, as slaves and slave_devs gets confusing in that context)

2. In few places Werror complained about broken aliasing, I've disabled Werror for now just to see if its worth for me to fix the warnings or not

3. Mellanox have a bunch of new defs that needs to be defined in configure_mlx5. Probably dummy config needs to be updated, but same as above.

4. A lot has been moved around in common/cnxk

5. rte_log seems to be required.

6. ntacc got removed, however can probably be copied back from old version

7. BNXT_TF_APP_ID seems to be required for bnxt driver

8. -mrtm or -mno-rtm is required as they use transactional memory in a few places.

I've also splitted my work into two commits - one to import dpdk and second one to port the patch (I did a diff with vanilla 23.03)

I kinda suspect that it is still a DPDK-related problem, but I'm not familiar enough with the code to build a minimal repro to report it to DPDK mail list though and it will take a bit of time to familiarize myself with the code base before I'll be able to do something like that...

[hhaim]

@Civil looking into the output of the TUI above, it seems there is an issue with the Tx counters too (port 0,1). Are you using CX-6 2x100Gbps with PCIe4 motherboard? we have CX-5de cards with PCIe3 motherboard in our lab, and you can't reach 100gbps total bandwidth due to PCI shortage. I would reduce the rate and look into the detail counters (in the Console show counters command to understand the reason for the error). We don't have CX-6 in our lab so we can't test it, usually upgrading the DPDK with NVIDIA driver is painful due to its dependency with kernel and OFED

[Civil]

@hhaim port 0 and 1 is a single cx-6 2x100, port 2 and 3 is cx5 2x100, cdat (pcie gen4 version) to show that it works with older generation just fine.

with PCIe4 motherboard?

Yes, both cards links at x16 Gen4.

We don't have CX-6 in our lab so we can't test it

I have CX-4, 5 and 6 on my hands, so I can run any kind of tests on any of those cards, if you have something in mind.

due to its dependency with kernel and OFED

If that matter, that machine of mine runs debian 12. I've tried OFED 5.7-1.0.2.0 and OFED 24.01-0.3.3.1, and haven't noticed any differences at all. As about kernel - I've tried 6.1 (stock) and 6.5 (backports)

And as I've said both trex 3.04 (dpdk 23.03) and trex after my attempt to port dpdk 23.11 shows same behavior. And same as original reporter said - trex 3.03 (dpdk 22.03) works fine.

I've so far tried to revert some of the changes around stats/xstats structure in drivers/net/mlx5/mlx5_stats.c for dpdk 23.03, but haven't observed any meaningful changes in the behavior (stats are still broken, just in another way).

If that matters, I've tried few firmware version on connectx6, currently I'm at 22.40.1000, but I've also tried 22.31.1014

[hhaim]

@Civil it seems that the trex driver counter code for CX-6 needs to be modified

try to understand using stats --xstats from the console to see the raw mlx5 driver counter in both CX-5 and CX-6

the trex mapping between the raw counter and trex counters is in this function

bool CTRexExtendedDriverBaseMlnx5G::get_extended_stats(CPhyEthIF * _if, CPhyEthIFStats *stats) {
    enum {

..

this function map raw_counter -> CPhyEthIFStats *stats after you will see the raw counters you will understand what is the issue (I hope)

[Civil]

In all cases I've started a 1kpps test before getting stats (start -f stl/bench.py -m 1kpps)

dpdk-23.11 variant:

trex>stats -x --port 1
Xstats

            Name:              |     Port 1:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |           42603
rx_good_bytes                  |             120
tx_good_bytes                  |         2556180
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |            7103
tx_q0_bytes                    |          426180
tx_q1_packets                  |            7100
tx_q1_bytes                    |          426000
tx_q2_packets                  |            7100
tx_q2_bytes                    |          426000
tx_q3_packets                  |            7100
tx_q3_bytes                    |          426000
tx_q4_packets                  |            7100
tx_q4_bytes                    |          426000
tx_q5_packets                  |            7100
tx_q5_bytes                    |          426000
rx_unicast_packets             |           42598
rx_unicast_bytes               |         2726272
tx_unicast_packets             |           42601
tx_unicast_bytes               |         2556060
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             128
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |           42603
rx_phy_packets                 |           42600
tx_phy_bytes                   |         2726592
rx_phy_bytes                   |         2726400

trex>stats -x --port 2

stats - Length of get_xstats_names: 79 and get_port_xstats_values: 63

Interesting enough

dpdk-23.03 variant (vanilla v3.04):

trex>stats -x --port 0
Xstats

            Name:              |     Port 0:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |           16348
rx_good_bytes                  |             120
tx_good_bytes                  |          980880
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |            2727
tx_q0_bytes                    |          163620
tx_q1_packets                  |            2725
tx_q1_bytes                    |          163500
tx_q2_packets                  |            2724
tx_q2_bytes                    |          163440
tx_q3_packets                  |            2724
tx_q3_bytes                    |          163440
tx_q4_packets                  |            2724
tx_q4_bytes                    |          163440
tx_q5_packets                  |            2724
tx_q5_bytes                    |          163440
rx_unicast_packets             |           16346
rx_unicast_bytes               |         1046144
tx_unicast_packets             |           16346
tx_unicast_bytes               |          980760
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             128
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |           16348
rx_phy_packets                 |           16348
tx_phy_bytes                   |         1046272
rx_phy_bytes                   |         1046272

trex>stats -x --port 2

stats - Length of get_xstats_names: 78 and get_port_xstats_values: 62

v3.03:

trex>stats -x --port 2
Xstats

            Name:              |     Port 2:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |           12715
rx_good_bytes                  |             120
tx_good_bytes                  |          762900
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |            2121
tx_q0_bytes                    |          127260
tx_q1_packets                  |            2119
tx_q1_bytes                    |          127140
tx_q2_packets                  |            2119
tx_q2_bytes                    |          127140
tx_q3_packets                  |            2119
tx_q3_bytes                    |          127140
tx_q4_packets                  |            2119
tx_q4_bytes                    |          127140
tx_q5_packets                  |            2118
tx_q5_bytes                    |          127080
rx_unicast_packets             |           12710
rx_unicast_bytes               |          762600
tx_unicast_packets             |           12713
tx_unicast_bytes               |          762780
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             120
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |           12715
rx_phy_packets                 |           12712
tx_phy_bytes                   |          813760
rx_phy_bytes                   |          813568

trex>stats -x --port 0
Xstats

            Name:              |     Port 0:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |           14424
rx_good_bytes                  |             120
tx_good_bytes                  |          865440
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |            2406
tx_q0_bytes                    |          144360
tx_q1_packets                  |            2404
tx_q1_bytes                    |          144240
tx_q2_packets                  |            2404
tx_q2_bytes                    |          144240
tx_q3_packets                  |            2404
tx_q3_bytes                    |          144240
tx_q4_packets                  |            2403
tx_q4_bytes                    |          144180
tx_q5_packets                  |            2403
tx_q5_bytes                    |          144180
rx_unicast_packets             |           14417
rx_unicast_bytes               |          922688
tx_unicast_packets             |           14421
tx_unicast_bytes               |          865260
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             128
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |           14423
rx_phy_packets                 |           14419
tx_phy_bytes                   |          923072
rx_phy_bytes                   |          922816

Also in attachments you'll find a screenshot from v3.03 (just in case that is anyhow helpful)

[Civil]

honestly I don't see anything wrong with raw counters there, but I'll have a look at the code later, maybe I'll find a problem there.

What makes me wonder is why stat -x is broken on cx5 and returns an error there, while in tui it is actually working fine.

[hhaim]

@Civil it seems that the TUI and counters work well in low 1kpps. try higher rates (in the older TUI snapshot it is about 10MPPS)

BTW the TUI works with trex-driver-counters this is the function that convert the (stats -x rte_eth_xstats_get) to trex counters. stats -x should work in CX-5

[Civil]

@hhaim pps rate doesn't change the behavior, even if I do just 1 pps it will be counted as ierrors in trex per-port stats table and in tui with dpdk 23.03+.

But here are the outputs for 10mpps test:

For dpdk-23.11:

trex>stats -x --port 0
Xstats

            Name:              |     Port 0:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |       101342080
rx_good_bytes                  |             120
tx_good_bytes                  |      6080524800
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |        16890899
tx_q0_bytes                    |      1013453940
tx_q1_packets                  |        16890928
tx_q1_bytes                    |      1013455680
tx_q2_packets                  |        16890896
tx_q2_bytes                    |      1013453760
tx_q3_packets                  |        16890096
tx_q3_bytes                    |      1013405760
tx_q4_packets                  |        16888813
tx_q4_bytes                    |      1013328780
tx_q5_packets                  |        16890448
tx_q5_bytes                    |      1013426880
rx_unicast_packets             |       101337174
rx_unicast_bytes               |      6485579136
tx_unicast_packets             |       101337206
tx_unicast_bytes               |      6080232360
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             128
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |       101335665
rx_phy_packets                 |       101335921
tx_phy_bytes                   |      6485490752
rx_phy_bytes                   |      6485498944

trex>stats -x --port 2

stats - Length of get_xstats_names: 79 and get_port_xstats_values:

v3.04 dpdk-23.03:

trex>stats -x --port 0
Xstats

            Name:              |     Port 0:
-------------------------------+----------------
rx_good_packets                |               2
tx_good_packets                |       154948775
rx_good_bytes                  |             120
tx_good_bytes                  |      9296926500
rx_q1_packets                  |               2
rx_q1_bytes                    |             120
tx_q0_packets                  |        25823013
tx_q0_bytes                    |      1549380780
tx_q1_packets                  |        25825640
tx_q1_bytes                    |      1549538400
tx_q2_packets                  |        25825352
tx_q2_bytes                    |      1549521120
tx_q3_packets                  |        25824773
tx_q3_bytes                    |      1549486380
tx_q4_packets                  |        25824741
tx_q4_bytes                    |      1549484460
tx_q5_packets                  |        25825256
tx_q5_bytes                    |      1549515360
rx_unicast_packets             |       154943838
rx_unicast_bytes               |      9916405632
tx_unicast_packets             |       154943912
tx_unicast_bytes               |      9296634720
rx_broadcast_packets           |               2
rx_broadcast_bytes             |             128
tx_broadcast_packets           |               2
tx_broadcast_bytes             |             120
tx_phy_packets                 |       154942240
rx_phy_packets                 |       154942496
tx_phy_bytes                   |      9916311552
rx_phy_bytes                   |      9916319744

trex>stats -x --port 2

stats - Length of get_xstats_names: 78 and get_port_xstats_values: 62

v3.03 (just stats -x as it can fetch stats for all ports without any problems, 2 and 3 is still cx5):

trex>stats -x
Xstats

            Name:              |     Port 0:     |     Port 1:     |     Port 2:     |     Port 3:
-------------------------------+-----------------+-----------------+-----------------+----------------
rx_good_packets                |               2 |               2 |               2 |               2
tx_good_packets                |        36135824 |        36170637 |        35699282 |        35726018
rx_good_bytes                  |             120 |             120 |             120 |             120
tx_good_bytes                  |      2168149440 |      2170238220 |      2141956920 |      2143561080
rx_q1_packets                  |               2 |               2 |               2 |               2
rx_q1_bytes                    |             120 |             120 |             120 |             120
tx_q0_packets                  |         6023141 |         6028976 |         5949904 |         5954328
tx_q0_bytes                    |       361388460 |       361738560 |       356994240 |       357259680
tx_q1_packets                  |         6023139 |         6028974 |         5949902 |         5954358
tx_q1_bytes                    |       361388340 |       361738440 |       356994120 |       357261480
tx_q2_packets                  |         6023011 |         6028781 |         5949869 |         5954325
tx_q2_bytes                    |       361380660 |       361726860 |       356992140 |       357259500
tx_q3_packets                  |         6023171 |         6028941 |         5949901 |         5954357
tx_q3_bytes                    |       361390260 |       361736460 |       356994060 |       357261420
tx_q4_packets                  |         6020192 |         6025992 |         5949837 |         5954293
tx_q4_bytes                    |       361211520 |       361559520 |       356990220 |       357257580
tx_q5_packets                  |         6023170 |         6028973 |         5949869 |         5954357
tx_q5_bytes                    |       361390200 |       361738380 |       356992140 |       357261420
rx_unicast_packets             |        36130891 |        36165758 |        35694160 |        35720992
rx_unicast_bytes               |      2312377024 |      2314608512 |      2141649600 |      2143259520
tx_unicast_packets             |        36130907 |        36165886 |        35694192 |        35721024
tx_unicast_bytes               |      2167854420 |      2169953160 |      2141651880 |      2143261440
rx_broadcast_packets           |               2 |               2 |               2 |               2
rx_broadcast_bytes             |             128 |             128 |             120 |             120
tx_broadcast_packets           |               2 |               2 |               2 |               2
tx_broadcast_bytes             |             120 |             120 |             120 |             120
tx_phy_packets                 |        36129197 |        36164285 |        35692609 |        35719378
rx_phy_packets                 |        36129485 |        36164477 |        35692786 |        35719506
tx_phy_bytes                   |      2312278848 |      2314521984 |      2284336256 |      2286048384
rx_phy_bytes                   |      2312287040 |      2314532672 |      2284338304 |      2286048384

stats -x should work in CX-5

For some reason it shows me the error I've showed you above. I haven't got time to dig into the function code yet, but at least I have a clue that something is wrong with how it is converted inside trex.

[Civil]

So about stats - Length of get_xstats_names: 78 and get_port_xstats_values: 62 my theory is that this error is not valid, because, quoting DPDK doc:

There is an assumption that 'xstat_names' and 'xstats' arrays are matched by array index: xstats_names[i].name => xstats[i].value

So that is expected when network card family can give more counters, than particular card supports so there could be more names than actual stats, index should still match.

I have a question - how performance critical is extended_stats method? Because what I want to try is to significantly simplify it but at the same try to make it future proof (e.x. getting rid of hardcoded enum list in favour of actual string comparisons in a first go, so if DPDK reshuffles resulting array it won't affect the result, etc.)

[Civil]

So I've rewrote extended_stats to be way simplier and after that it works well on both ConnectX-5 and 6.

That is what I did: https://github.com/Civil/trex-core/commit/a22596ac0b67ad6b9fac02c7a979332547f16e37

And if you are interested, I can port it to current master and send a PR for that.

It might require a bit of extra work as I just made sure that my use case works well, and I probably haven't got enough historical context about some of the decisions, but it can be a good start for you.

[hhaim]

@Civil

nice.

looked into your commit, it is strange that some counters do not work in diffs (e.g. ibytes/obytes/errors)

for example:

 stats->ibytes = ibytes +
                         packet_diff * 4; // add FCS

was that due to a kernel driver change?

If you can create a PR I could run it in our regression to verify it on our hardware

[Civil]

was that due to a kernel driver change?

That was because I've just removed the code (kept only names of the counters) and rewrote it from scratch and haven't understood the value of getting diff.

I can return it, but it only make sense for moments when driver counter wraps (and that I assumed happens only if it is actually not uint64 but something else underneath, and I haven't seen any comments about that inside DPDK code), so in the PR I can keep the diffs back in place, they shouldn't hurt as I think main benefit is getting a single reliable way to get the counter, without relying on their exact position in the array (and that is what has changed in DPDK)

[hhaim]

@Civil understood, it is better to keep diff for all the counters, it does not relate to the wrap (wrap is another issue). otherwise, you will have a huge jump in the first read. all the counters are 64bit and going up.

[Civil]

Ok, understood; thanks for the clarification. I've did a PR that keeps the diff: https://github.com/cisco-system-traffic-generator/trex-core/pull/1106

P.S. I've also verified that same problem happened on BlueField-2 without the PR and PR fixes that as well (which is expected I guess, because it is essentially ConnectX-6 with extras).

And in case you need more tests or more information I have in my homelab both CX6 and BF2 now.

[Civil]

@norg I think after my PR was accepted, trex (from master or if you manually apply the patch from PR) would work for you as well.

At least I can now get everything working even with mix of different ConnectX cards.

[norg]

@norg I think after my PR was accepted, trex (from master or if you manually apply the patch from PR) would work for you as well.

At least I can now get everything working even with mix of different ConnectX cards.

Awesome, sounds great. Thanks!