Unable to collect flow stats and latency stats on E810 card on TRex 3.0.0

[MaciejWachowski]

I am unable to collect flow stats and latency stats on E810 card on TRex 3.0.0. When I am trying to collect stats I get following output:

{'global': {'active_flows': 0.0, 'active_sockets': 0, 'bw_per_core': 20.25494956970215, 'cpu_util': 4.858463764190674,
            'cpu_util_raw': 4.2580647468566895, 'open_flows': 0.0, 'platform_factor': 1.0, 'rx_bps': 30506491904.0,
            'rx_core_pps': 0.0, 'rx_cpu_util': 0.0, 'rx_drop_bps': 0.0, 'rx_pps': 59582976.0, 'socket_util': 0.0,
            'tx_expected_bps': 0.0, 'tx_expected_cps': 0.0, 'tx_expected_pps': 0.0, 'tx_pps': 59582920.0,
            'tx_bps': 30506461184.0, 'tx_cps': 0.0, 'total_servers': 0, 'total_clients': 0, 'total_alloc_error': 0,
            'queue_full': 0},
 0: {'opackets': 892857144, 'ipackets': 892857144, 'obytes': 57142857216, 'ibytes': 57142857216, 'oerrors': 0,
     'ierrors': 0, 'tx_bps': 7626593280.0, 'tx_pps': 14895686.0, 'tx_bps_L1': 10009903040.0, 'tx_util': 10.00990304,
     'rx_bps': 7626584064.0, 'rx_pps': 14895717.0, 'rx_bps_L1': 10009898784.0, 'rx_util': 10.009898784},
 1: {'opackets': 892857144, 'ipackets': 892857144, 'obytes': 57142857216, 'ibytes': 57142857216, 'oerrors': 0,
     'ierrors': 0, 'tx_bps': 7626629632.0, 'tx_pps': 14895754.0, 'tx_bps_L1': 10009950272.0, 'tx_util': 10.009950272,
     'rx_bps': 7626637824.0, 'rx_pps': 14895762.0, 'rx_bps_L1': 10009959744.0, 'rx_util': 10.009959744},
 'total': {'opackets': 3571428576, 'ipackets': 3571428576, 'obytes': 228571428864, 'ibytes': 228571428864, 'oerrors': 0,
           'ierrors': 0, 'tx_bps': 30506462208.0, 'tx_pps': 59582922.0, 'tx_bps_L1': 40039729728.0,
           'tx_util': 40.039729728, 'rx_bps': 30506492416.0, 'rx_pps': 59582978.0, 'rx_bps_L1': 40039768896.0,
           'rx_util': 40.039768896},
 'flow_stats': {'global': {'rx_err': {0: 0, 1: 0, 2: 0, 3: 0}, 'tx_err': {0: 0, 1: 0, 2: 0, 3: 0}},
                0: {'rx_pkts': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0}, 'rx_bytes': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0},
                    'tx_pkts': {0: 892857084, 1: 0, 2: 0, 3: 0, 'total': 892857084},
                    'tx_bytes': {0: 57142853376, 1: 0, 2: 0, 3: 0, 'total': 57142853376},
                    'rx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'rx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'rx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0}},
                1: {'rx_pkts': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0}, 'rx_bytes': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0},
                    'tx_pkts': {0: 0, 1: 892857084, 2: 0, 3: 0, 'total': 892857084},
                    'tx_bytes': {0: 0, 1: 57142853376, 2: 0, 3: 0, 'total': 57142853376},
                    'rx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'rx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'rx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                    'tx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0}},
                10: {'rx_pkts': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0}, 'rx_bytes': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0},
                     'tx_pkts': {0: 60, 1: 0, 2: 0, 3: 0, 'total': 60},
                     'tx_bytes': {0: 3840, 1: 0, 2: 0, 3: 0, 'total': 3840},
                     'rx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'rx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'rx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0}},
                11: {'rx_pkts': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0}, 'rx_bytes': {0: 0, 1: 0, 2: 0, 3: 0, 'total': 0},
                     'tx_pkts': {0: 0, 1: 60, 2: 0, 3: 0, 'total': 60},
                     'tx_bytes': {0: 0, 1: 3840, 2: 0, 3: 0, 'total': 3840},
                     'rx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'rx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_bps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_pps': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'rx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0},
                     'tx_bps_l1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 'total': 0.0}},
                },
 'latency': {'global': {'old_flow': 0, 'bad_hdr': 0},
             10: {'err_cntrs': {'dropped': 0, 'out_of_order': 0, 'dup': 0, 'seq_too_high': 0, 'seq_too_low': 0},
                  'latency': {'jitter': 0, 'average': 0.0, 'total_max': 0, 'total_min': 0, 'last_max': 0,
                              'histogram': {}}},
             11: {'err_cntrs': {'dropped': 0, 'out_of_order': 0, 'dup': 0, 'seq_too_high': 0, 'seq_too_low': 0},
                  'latency': {'jitter': 0, 'average': 0.0, 'total_max': 0, 'total_min': 0, 'last_max': 0,
                              'histogram': {}}}
}

Counters for flow stats and latency are zeros

[anthonyfee]

I am also seeing this issue. It would be great to know if there is a patch or something that we could apply to resolve the issue?

[hhaim]

@anthonyfee this is a known DPDK driver issue, try to move to software mode using --software in CLI

[anthonyfee]

@hhaim Thank you for the response. I'll give this a try!

[anthonyfee]

@hhaim This worked, thanks again for the suggestion. Do you know if it possible to change the software option without restarting T-Rex? I'm interfacing through the Python API and it would be nice to disable/enable depending on the type of test that I am running. I can restart T-Rex but that's a bit slow. Thanks!

[hhaim]

@anthonyfee you need to restart.

[anthonyfee]

@hhaim Good to know, thanks again.