search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
739 stars 1.43k forks source link

[Z9332] : RX Drop is not accounted in interface counters when PFCWD is triggered in bidirectional traffic. #9193

Closed chitra-raghavan closed 3 years ago

chitra-raghavan commented 3 years ago

Description

Test Setup

PFCWD is enabled on ports Ethernet0 and Ethernet2 Bi directional traffic is sent from ports Ethernet0 and Ethernet2 with Priority P3. PFC P3 pause is sent from IXIA connected to Ethernet0 .

Describe the results you received:

When PFC pause is received on Ethernet0, the port is stormed. Packets are dropped at RX in port Ethernt0 and at Tx in Ethernet2. Broadcom RDGB counters shows the drops , But the RX Drop counters are not shown in "show interface counters" as well as " show pfcwd stats" 

root@sonic-10431:~# show pfcwd config
Changed polling interval to 100ms
     PORT    ACTION    DETECTION TIME    RESTORATION TIME
---------  --------  ----------------  ------------------
Ethernet0      drop               200                 200
Ethernet2      drop               200                 200
root@sonic-10431:~#

root@sonic-10431:~# cat show_int_counter
      IFACE    STATE       RX_OK       RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR       TX_OK       TX_BPS    TX_UTIL    TX_ERR     TX_DRP    TX_OVR
  Ethernet0        U  61,115,381  286.58 KB/s      0.00%         0         0         0  51,847,244  233.95 KB/s      0.00%         0  9,268,210         0
  Ethernet2        U  61,115,379  280.69 KB/s      0.00%         0         0         0  51,847,202  233.91 KB/s      0.00%         0          0         0
  Ethernet4        U           6     0.00 B/s      0.00%         0         0         0          74     3.05 B/s      0.00%         0          0         0

root@sonic-10431:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet0:3  operational                        1/1     0/9268210  2338/9268248          0/9268210       2338/9268248
root@sonic-10431:~#
root@sonic-10431:~# show c ce0
RUC_64.ce0                  :            61,115,380          +8,451,799
RDBGC0_64.ce0               :             9,268,248          +1,408,633
RDBGC4_64.ce0               :             9,268,248          +1,408,633
TDBGC3_64.ce0               :             9,268,210          +1,408,632
CDMIB_R255.ce0              :             9,270,586          +1,408,631
CDMIB_R1518.ce0             :            61,115,376          +8,451,786
CDMIB_R64.ce0               :                     6                  +1
CDMIB_RPRM.ce0              :            61,115,380          +8,451,787
CDMIB_T127.ce0              :                    18                  +4
CDMIB_T1518.ce0             :            51,847,166          +7,043,155
CDMIB_T64.ce0               :                    19                  +3
CDMIB_T255.ce0              :                    51                  +9
CDMIB_RPFC3.ce0             :             9,270,586          +1,408,631
CDMIB_RPKT.ce0              :            70,385,968          +9,860,418
CDMIB_RUCA.ce0              :            61,115,380          +8,451,787
CDMIB_RXCF.ce0              :             9,270,586          +1,408,631
CDMIB_RPOK.ce0              :            61,115,382          +8,451,787
CDMIB_RXPP.ce0              :             9,270,586          +1,408,631
CDMIB_TBCA.ce0              :                    14                  +2
CDMIB_TBYT.ce0              :        53,091,513,513      +7,212,193,695
CDMIB_RBYT.ce0              :        63,768,780,416      +8,834,933,696
CDMIB_RPROG0.ce0            :                     6                  +1
CDMIB_TMCA.ce0              :                    69                 +13
CDMIB_TPOK.ce0              :            51,847,254          +7,043,171
CDMIB_TPKT.ce0              :            51,847,254          +7,043,171
CDMIB_TUCA.ce0              :            51,847,171          +7,043,156
MC_PERQ_PKT(3).ce0          :                    10                  +1
MC_PERQ_BYTE(3).ce0         :                   640                 +64
UC_PERQ_PKT(0).ce0          :            10,185,896          +1,408,264
UC_PERQ_PKT(1).ce0          :            10,185,896          +1,408,264
UC_PERQ_PKT(2).ce0          :            10,185,896          +1,408,264
UC_PERQ_PKT(5).ce0          :            10,185,896          +1,408,264
UC_PERQ_PKT(6).ce0          :            10,185,896          +1,408,264
UC_PERQ_PKT(7).ce0          :                    78                 +15
UC_PERQ_BYTE(0).ce0         :        10,430,357,504      +1,442,062,336
UC_PERQ_BYTE(1).ce0         :        10,430,357,504      +1,442,062,336
UC_PERQ_BYTE(2).ce0         :        10,430,357,504      +1,442,062,336
UC_PERQ_BYTE(5).ce0         :        10,430,357,504      +1,442,062,336
UC_PERQ_BYTE(6).ce0         :        10,430,357,504      +1,442,062,336
UC_PERQ_BYTE(7).ce0         :                14,889              +2,911
drivshell>
root@sonic-10431:~#
root@sonic-10431:~# show  dropcounters  counts
      IFACE    STATE    RX_ERR    RX_DROPS    TX_ERR    TX_DROPS
-----------  -------  --------  ----------  --------  ----------
  Ethernet0        U         0           0         0     9268210
  Ethernet2        U         0           0         0           0
  Ethernet4        U         3           0         0           0

Describe the results you expected:

RX Drops should be should be incremented in Show interface counters and pfcwd counters

Output of show version:

root@sonic-10431:~# show ver

SONiC Software Version: SONiC.202012.49263-8ada00630
Distribution: Debian 10.11
Kernel: 4.19.0-12-2-amd64
Build commit: 8ada00630
Build date: Sun Nov  7 14:52:53 UTC 2021
Built by: AzDevOps@sonic-build-workers-000VAX

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH04CN21CET009BR0026
Uptime: 22:09:05 up  8:37,  1 user,  load average: 0.58, 0.88, 1.00

Docker images:
REPOSITORY                    TAG                      IMAGE ID            SIZE
docker-sonic-mgmt-framework   202012.49263-8ada00630   626220ba09a5        815MB
docker-sonic-mgmt-framework   latest                   626220ba09a5        815MB
docker-sonic-telemetry        202012.49263-8ada00630   6dbd8e3e4cf5        492MB
docker-sonic-telemetry        latest                   6dbd8e3e4cf5        492MB
docker-orchagent              202012.49263-8ada00630   1aafb9ec1e9d        431MB
docker-orchagent              latest                   1aafb9ec1e9d        431MB
docker-fpm-frr                202012.49263-8ada00630   ee895f19d681        431MB
docker-fpm-frr                latest                   ee895f19d681        431MB
docker-sflow                  202012.49263-8ada00630   0795386d5b79        414MB
docker-sflow                  latest                   0795386d5b79        414MB
docker-teamd                  202012.49263-8ada00630   7c6788fb08b7        413MB
docker-teamd                  latest                   7c6788fb08b7        413MB
docker-nat                    202012.49263-8ada00630   2e889de82e1c        416MB
docker-nat                    latest                   2e889de82e1c        416MB
docker-platform-monitor       202012.49263-8ada00630   647d884edff0        614MB
docker-platform-monitor       latest                   647d884edff0        614MB
docker-snmp                   202012.49263-8ada00630   f25f22475532        444MB
docker-snmp                   latest                   f25f22475532        444MB
docker-syncd-brcm             202012.49263-8ada00630   3105e051a93a        695MB
docker-syncd-brcm             latest                   3105e051a93a        695MB
docker-lldp                   202012.49263-8ada00630   304acda31e5e        443MB
docker-lldp                   latest                   304acda31e5e        443MB
docker-dhcp-relay             202012.49263-8ada00630   314e266c5d18        416MB
docker-dhcp-relay             latest                   314e266c5d18        416MB
docker-database               202012.49263-8ada00630   79e5bdc68eec        402MB
docker-database               latest                   79e5bdc68eec        402MB
docker-router-advertiser      202012.49263-8ada00630   72d04f78f0d6        403MB
docker-router-advertiser      latest                   72d04f78f0d6        403MB

root@sonic-10431:~#

Steps to reproduce the issue:

  1. Send Bi directional P3 traffic from 2 ports
  2. Send PFC P3 pause from from one of the port
  3. Check for Rx and Tx drops in show interface counter and show pfc stats

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

gechiang commented 3 years ago

@chitra-raghavan Can you you try removing the SOC property "sai_adjust_acl_drop_in_rx_drop=1" from the corresponding z9332f config.bcm file such as: device/dell/x86_64-dellemc_z9332f_d1508-r0/DellEMC-Z9332f-M-O16C64/th3-z9332f-16x400G-64x100G.config.bcm device/dell/x86_64-dellemc_z9332f_d1508-r0/DellEMC-Z9332f-O32/th3-z9332f-32x400G.config.bcm To narrow down if the behavior you observed is due to this SOC property enabled which will stop counting any ACL related drops from the show interface counters "RXDRP" count... Thanks!

chitra-raghavan commented 3 years ago

@gechiang , After commenting the SOC property "sai_adjust_acl_drop_in_rx_drop=1" from the corresponding z9332f config.bcm file , the ingress drops are shown in show interface counters .


root@sonic-10431:~# show interfaces  counters | grep  U
      IFACE    STATE      RX_OK       RX_BPS    RX_UTIL    RX_ERR     RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR     TX_DRP    TX_OVR
  Ethernet0        U  1,550,317  215.01 KB/s      0.00%         0  1,247,969         0  302,413    1.15 B/s      0.00%         0  1,247,912         0
  Ethernet2        U  1,550,312  191.10 KB/s      0.00%         0          0         0  302,351    9.92 B/s      0.00%         0          0         0
  Ethernet4        U          0     0.00 B/s      0.00%         0          0         0        8   17.96 B/s      0.00%         0          0         0
  Ethernet6        U          0     0.00 B/s      0.00%         0          0         0        8   17.96 B/s      0.00%         0          0         0
 Ethernet96        U        218   118.79 B/s      0.00%         0          0         0      168  117.40 B/s      0.00%         0          0         0
root@sonic-10431:~#
root@sonic-10431:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet0:4  operational                        1/1     0/1247912  8637/1247695          0/1247912       8637/1247695

root@sonic-10431:~#
vmittal-msft commented 3 years ago

As per debug call today morning, here are the findings -

  1. PFCWD behavior is working fine with or without SOC property "sai_adjust_acl_drop_in_rx_drop=1" part of config.bcm file.
  2. This SOC property was introduced to ignore RxDrops due to ACL (installed by user) as part of interface counters. Since PFCWD logic install ACL to drop Tx/Rx traffic during storm. it is not counted as part of interface counters.
  3. show pfcwd stats is correctly showing drop count for Rx/Tx direction as per chip counters.

Since functionality is working fine, will close this PR.