search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
722 stars 1.38k forks source link

[Z9332][202012]: PFC WD is not working #8483

Closed chitra-raghavan closed 2 years ago

chitra-raghavan commented 3 years ago

Description

[Z9332][202012]: PFC WD is not working in 202012 images. This is seen after inclusion of PR https://github.com/Azure/sonic-swss/pull/1111 in 202012 branch. Without the PR changes ( with older version of pfc_detect_broadcom.lua) , PFCwd works fine.

Steps to reproduce the issue:

  1. Enable PFCwd with drop action
  2. Send P3 Traffic from Port 1 to port2
  3. Send PFC pause for P3 from Port2 to port1 .

Describe the results you received:

in Latest image

When PFC pause frames are received, instead of dropping packets at the device , the PFC pause frames are transmitted to the Ingress port . 

root@sonic-10431:~# show pfcwd config
Changed polling interval to 100ms
     PORT    ACTION    DETECTION TIME    RESTORATION TIME
---------  --------  ----------------  ------------------
Ethernet0      drop               200                 200
Ethernet4      drop               200                 200
root@sonic-10431:~#

root@sonic-10431:~# show pfcwd  stats
  QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-------  --------  -------------------------  ------------  ------------  -----------------  -----------------

P3 Traffic counters 

root@sonic-10431:~#
root@sonic-10431:~# show queue counters  Ethernet4
Last cached time was 2021-08-01 01:09:20.460621
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet4    UC0               0                0            0             0
Ethernet4    UC1               0                0            0             0
Ethernet4    UC2               0                0            0             0
Ethernet4    UC3      18,503,169   17,805,683,712            0             0
Ethernet4    UC4               0                0            0             0
Ethernet4    UC5               0                0            0             0
Ethernet4    UC6               0                0            0             0
Ethernet4    UC7               8            1,907            0             0
Ethernet4    MC8               0                0            0             0
Ethernet4    MC9               0                0            0             0
Ethernet4   MC10               0                0            0             0
Ethernet4   MC11               0                0            0             0

root@sonic-10431:~# show interfaces  counters  | grep U
      IFACE    STATE       RX_OK        RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR       TX_OK        TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
  Ethernet0        U  32,364,375  1232.13 MB/s      9.86%         0         0         0           1      9.48 B/s      0.00%         0         0         0
  Ethernet2        U           0      0.00 B/s      0.00%         0         0         0           1      9.48 B/s      0.00%         0         0         0
  Ethernet4        U           0      0.00 B/s      0.00%         0         0         0  32,364,402  1232.13 MB/s      9.86%         0         0         0
  Ethernet6        U           0      0.00 B/s      0.00%         0         0         0           1      9.48 B/s      0.00%         0         0         0
Ethernet112        U           0      0.00 B/s      0.00%         0         0         0          27    133.47 B/s      0.00%         0         0         0
root@sonic-10431:~#

PFC pause frames received at egress port : 

root@sonic-10431:~# show pfc counters  | grep Rx -A 10
    Port Rx    PFC0    PFC1    PFC2        PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ----------  ------  ------  ------  ------
  Ethernet0       0       0       0           0       0       0       0       0
  Ethernet2       0       0       0           0       0       0       0       0
  Ethernet4       0       0       0  83,064,491       0       0       0       0
  Ethernet6       0       0       0           0       0       0       0       0

PFC packets sent out 

root@sonic-10431:~#
root@sonic-10431:~# show pfc counters  | grep Tx -A 5
    Port Tx    PFC0    PFC1    PFC2     PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  -------  ------  ------  ------  ------
  Ethernet0       0       0       0  399,485       0       0       0       0
  Ethernet2       0       0       0        0       0       0       0       0
  Ethernet4       0       0       0        0       0       0       0       0
  Ethernet6       0       0       0        0       0       0       0       0

With older version of pfc_detect_broadcom.lua 

root@sonic-10431:~# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        1/0    0/41507426           0/0         0/41507426                0/0
root@sonic-10431:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        1/1    0/45830961           0/0         0/45830961                0/0
root@sonic-10431:~# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        2/1    0/46497254           0/0           0/666293                0/0
root@sonic-10431:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        2/2    0/52297325           0/0          0/6466364                0/0
root@sonic-10431:~# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        3/2    0/55746196           0/0          0/3448871                0/0
root@sonic-10431:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        3/3    0/55746196           0/0          0/3448871                0/0

root@sonic-10431:~# show interfaces  counters  | grep U
      IFACE    STATE       RX_OK       RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR       TX_OK       TX_BPS    TX_UTIL    TX_ERR      TX_DRP    TX_OVR
  Ethernet0        U  83,275,694  906.42 MB/s      7.25%         0         0         0           3  2194.57 B/s      0.00%         0           0         0
  Ethernet2        U           0     0.00 B/s      0.00%         0         0         0           3     8.13 B/s      0.00%         0           0         0
  Ethernet4        U           0   63.72 MB/s      0.51%         0         0         0  26,200,069  285.18 MB/s      2.28%         0  57,075,628         0
  Ethernet6        U           0     0.00 B/s      0.00%         0         0         0           3     8.13 B/s      0.00%         0           0         0
Ethernet112        U           0     0.00 B/s      0.00%         0         0         0          97   136.25 B/s      0.00%         0           0         0
root@sonic-10431:~#
root@sonic-10431:~#

root@sonic-10431:~# show pfc counters  | grep Tx -A5
    Port Tx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0   3,214       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0       0       0       0       0       0
  Ethernet6       0       0       0       0       0       0       0       0
root@sonic-10431:~# show pfc counters  | grep Rx -A5
    Port Rx    PFC0    PFC1    PFC2        PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ----------  ------  ------  ------  ------
  Ethernet0       0       0       0           0       0       0       0       0
  Ethernet2       0       0       0           0       0       0       0       0
  Ethernet4       0       0       0  46,829,555       0       0       0       0
  Ethernet6       0       0       0           0       0       0       0       0
root@sonic-10431:~#

Describe the results you expected:

Output of show version:

root@sonic-10431:~# show ver

SONiC Software Version: SONiC.202012.28488-ada56abe6
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: ada56abe6
Build date: Wed Aug 11 17:44:34 UTC 2021
Built by: AzDevOps@sonic-build-workers-000L3W

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH04CN21CET009BR0026
Uptime: 01:51:15 up  2:09,  2 users,  load average: 1.08, 0.98, 0.93

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

chitra-raghavan commented 3 years ago

Atatched redis-cli monitor and redis-dump for db2 as requested

redis-dump_db2.txt redis-monitor.txt

chitra-raghavan commented 2 years ago

Fixed in latest 202012 images