search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
734 stars 1.41k forks source link

[202012][Z9332][Pfcwd + Asym Pfc]: With Asym PFC enabled and when pfcwd storm is detected on priority P3 and P4 in a port , PFC packets egress out only for one Priority #9453

Open chitra-raghavan opened 2 years ago

chitra-raghavan commented 2 years ago

Description

With Asym PFC enabled and when pfcwd storm is detected on priority 3 and 4 in a port , PFC packets egress out only for one Priority

Steps to reproduce the issue:

Test Setup :

  1. Connect 2 IXIA ports to DUT ( Ethernet4 and Ethernet6)
  2. Configure PFCWD drop action on port P2 ( Ethernet6)
  3. Configure Asymmetric pfc on port P2 (Ethernet6)
  4. Send P3 and P4 Data traffic from port P1 to P2 (Ethernet4 to Ethernet6)
  5. Send PFC pause frames for Priorities P3 and P4 from port P2 (Ethernet6)

Describe the results you received:

  1. Working scenario: Pause for P3 and P4 are sent at different time instance : PFC storm is detected for corresponding priority
    PFC pause frames egress out through port P1 for that corresponding priority as expected

  2. Issue scenario: Both P3 and P4 pause frames are sent at same time PFCWD storm is detected for both priorities . But PFC frames egress out only for one priority.

    Send PFC Packets for both priorites P3 and P4

root@sonic-10432:/home/admin# show queue  counters Ethernet6
Last cached time was 2021-12-07 14:10:05.528068
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet6    UC0               0                0            0             0
Ethernet6    UC1               0                0            0             0
Ethernet6    UC2               0                0            0             0
Ethernet6    UC3          56,168      102,683,648            0             0
Ethernet6    UC4          56,168      102,682,624            0             0
Ethernet6    UC5               0                0            0             0
Ethernet6    UC6               0                0            0             0
Ethernet6    UC7               0                0            0             0
Ethernet6    MC8               0                0            0             0
Ethernet6    MC9               0                0            0             0
Ethernet6   MC10               0                0            0             0
Ethernet6   MC11               0                0            0             0

root@sonic-10432:/home/admin#

root@sonic-10432:~# show interfaces  status Ethernet6
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin                                             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  -----------------------------------------------  ----------
  Ethernet6    39,40     100G   9100     rs    etp1d   trunk      up       up  QSFP-DD Double Density 8X Pluggable Transceiver          on
root@sonic-10432:~# show pfcwd config
Changed polling interval to 200ms
      PORT    ACTION    DETECTION TIME    RESTORATION TIME
----------  --------  ----------------  ------------------
 Ethernet4      drop               200                 200
 Ethernet6      drop               200                 200
Ethernet96      drop               200                 200
root@sonic-10432:~# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP      RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  --------------  -----------------  -----------------
Ethernet4:3  operational                        2/2    0/22408450  16401/22407182         0/22354649      9020/22354304
Ethernet6:3      stormed                        3/2       0/46043      3410/44949                0/0                0/0
Ethernet6:4      stormed                        3/2           0/0             0/0                0/0                0/0
root@sonic-10432:~#

Check PFC counters

The PFC frames are received for both priorities at same time , PFC packets egress out of sonic DUT for only one priority

root@sonic-10432:/home/admin# show pfc counters | grep Rx -A10
    Port Rx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0       0       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0       0       0       0       0       0
  Ethernet6       0       0       0  93,845  93,844       0       0       0
  Ethernet8       0       0       0       0       0       0       0       0
 Ethernet10       0       0       0       0       0       0       0       0
 Ethernet12       0       0       0       0       0       0       0       0
 Ethernet14       0       0       0       0       0       0       0       0
 Ethernet16       0       0       0       0       0       0       0       0
root@sonic-10432:/home/admin#

root@sonic-10432:/home/admin# show pfc counters | grep Tx -A10
    Port Tx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0       0       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0  63,795       0       0       0       0
  Ethernet6       0       0       0       0       0       0       0       0
  Ethernet8       0       0       0       0       0       0       0       0
 Ethernet10       0       0       0       0       0       0       0       0
 Ethernet12       0       0       0       0       0       0       0       0
 Ethernet14       0       0       0       0       0       0       0       0
 Ethernet16       0       0       0       0       0       0       0       0
root@sonic-10432:/home/admin#

Stop P3 pause , Now P4 PFC packets egress out of Port Ethernet4 and vice versa. 

root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP      RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  --------------  -----------------  -----------------
Ethernet4:3  operational                        2/2    0/22408450  16401/22407182         0/22354649      9020/22354304
Ethernet6:3  operational                        4/4       0/46043      3410/44949                0/0                0/0
Ethernet6:4      stormed                        4/3           0/0             0/0                0/0                0/0
root@sonic-10432:/home/admin#

root@sonic-10432:/home/admin# show pfc counters | grep Rx -A5
    Port Rx    PFC0    PFC1    PFC2       PFC3        PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ---------  ----------  ------  ------  ------
  Ethernet0       0       0       0          0           0       0       0       0
  Ethernet2       0       0       0          0           0       0       0       0
  Ethernet4       0       0       0          0           0       0       0       0
  Ethernet6       0       0       0  7,656,732  10,096,878       0       0       0
root@sonic-10432:/home/admin#
root@sonic-10432:/home/admin# show pfc counters | grep Rx -A5
    Port Rx    PFC0    PFC1    PFC2       PFC3        PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ---------  ----------  ------  ------  ------
  Ethernet0       0       0       0          0           0       0       0       0
  Ethernet2       0       0       0          0           0       0       0       0
  Ethernet4       0       0       0          0           0       0       0       0
  Ethernet6       0       0       0  7,656,732  10,224,362       0       0       0
root@sonic-10432:/home/admin# show pfc counters | grep Tx -A5
    Port Tx    PFC0    PFC1    PFC2       PFC3     PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ---------  -------  ------  ------  ------
  Ethernet0       0       0       0          0        0       0       0       0
  Ethernet2       0       0       0          0        0       0       0       0
  Ethernet4       0       0       0  1,012,785  355,540       0       0       0
  Ethernet6       0       0       0          0        0       0       0       0
root@sonic-10432:/home/admin# show pfc counters | grep Tx -A5
    Port Tx    PFC0    PFC1    PFC2       PFC3     PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ---------  -------  ------  ------  ------
  Ethernet0       0       0       0          0        0       0       0       0
  Ethernet2       0       0       0          0        0       0       0       0
  Ethernet4       0       0       0  1,012,785  365,158       0       0       0
  Ethernet6       0       0       0          0        0       0       0       0
root@sonic-10432:/home/admin#

Describe the results you expected:

PFC packets should get triggered for both the priorities

Output of show version:

root@sonic-10432:/home/admin# show ver

SONiC Software Version: SONiC.20201231.44
Distribution: Debian 10.11
Kernel: 4.19.0-12-2-amd64
Build commit: 63ef7f53bc
Build date: Wed Nov 24 02:53:38 UTC 2021
Built by: cloudtest@876d3befc000000

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH0J5HDGCET0094A0009
Uptime: 14:23:57 up  2:52,  1 user,  load average: 0.79, 1.20, 1.09

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-brcm          20201231.44         9f91a09d0e9c        672MB
docker-syncd-brcm          latest              9f91a09d0e9c        672MB
docker-teamd               20201231.44         48053e1d92e8        390MB
docker-teamd               latest              48053e1d92e8        390MB
docker-router-advertiser   20201231.44         b6f6ccc4ba7f        380MB
docker-router-advertiser   latest              b6f6ccc4ba7f        380MB
docker-platform-monitor    20201231.44         76c085c0a328        561MB
docker-platform-monitor    latest              76c085c0a328        561MB
docker-lldp                20201231.44         a4a5ad2cde5e        420MB
docker-lldp                latest              a4a5ad2cde5e        420MB
docker-snmp                20201231.44         2439767a3279        422MB
docker-snmp                latest              2439767a3279        422MB
docker-dhcp-relay          20201231.44         aee1d0cc3dbb        393MB
docker-dhcp-relay          latest              aee1d0cc3dbb        393MB
docker-database            20201231.44         da24c051253b        379MB
docker-database            latest              da24c051253b        379MB
docker-orchagent           20201231.44         820bdc795273        408MB
docker-orchagent           latest              820bdc795273        408MB
docker-sonic-telemetry     20201231.44         178c953e9f05        469MB
docker-sonic-telemetry     latest              178c953e9f05        469MB
docker-mux                 20201231.44         bcd196bd7a3c        432MB
docker-mux                 latest              bcd196bd7a3c        432MB
docker-fpm-frr             20201231.44         0e266910f4b7        408MB
docker-fpm-frr             latest              0e266910f4b7        408MB
docker-sonic-restapi       20201231.44         6b791c7e73e3        345MB
docker-sonic-restapi       latest              6b791c7e73e3        345MB
docker-acms                20201231.44         d0848884cc78        181MB
docker-acms                latest              d0848884cc78        181MB
k8s.gcr.io/pause           3.4.1               0f8457a4c2ec        683kB

root@sonic-10432:/home/admin#

Output of show techsupport:

sonic_dump_sonic-10432_20211207_141933.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

zhangyanzhao commented 2 years ago

Feature gap and addressed by PR https://github.com/Azure/sonic-swss/pull/1887