search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
724 stars 1.38k forks source link

[Z9332 Th3]: When Multiple PFC P3/P4 pause frames are started and stopped simultaneously, PFC Packets are sent out from PFCWD configured ports #9430

Open chitra-raghavan opened 2 years ago

chitra-raghavan commented 2 years ago

Description

When Multiple PFC P3/P4 pause frames are started and stopped simultaneously, PFC Packets are sent out from PFCWD configured ports

Device : Z9332 ASIC : Th3 SKU : DellEMC-Z9332f-M-O16C64

Steps to reproduce the issue:

Test Setup :

  1. Connect 2 (or more) IXIA ports to DUT
  2. Configure PFCWD drop action on both the ports
  3. Send bidirectional P3 Data Traffic ( say 10000 Packets/sec)
  4. Start P3 pause frames from both the ports on P3 simultaneously.
  5. Stop both pause frames after a while.
  6. Stop Data traffic

Describe the results you received:

When pause frames are stopped from IXIA, Could see PFC packets ae triggered out randomly on either of the ports, Even after stopping data traffic, PFC frames are sent out from the Device .

Before sending pause frames : Traffic is sent on ports Ethernet4 and Ethernet6 

root@sonic-10432:/home/admin# show pfcwd config
Changed polling interval to 200ms
      PORT    ACTION    DETECTION TIME    RESTORATION TIME
----------  --------  ----------------  ------------------
 Ethernet4      drop               200                 200
 Ethernet6      drop               200                 200
Ethernet96      drop               200                 200
root@sonic-10432:/home/admin#

root@sonic-10432:/home/admin# show interfaces  counters  | grep U
show       IFACE    STATE    RX_OK        RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK        TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
  Ethernet2        U        0      0.00 B/s      0.00%         0         0         0        1     13.39 B/s      0.00%         0         0         0
  Ethernet4        U   98,880  8767.17 KB/s      0.07%         0         0         0   98,878  8767.06 KB/s      0.07%         0         0         0
  Ethernet6        U   98,941  8777.42 KB/s      0.07%         1         0         0   98,944  8777.65 KB/s      0.07%         0         0         0
Ethernet104        U        1      3.76 B/s      0.00%         0         0         0       21    141.80 B/s      0.00%         0         0         0
proot@sonic-10432:/home/admin# 

root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        1/1       0/56035    5419/55139            0/56035         5419/55139
Ethernet6:3  operational                        1/1       0/46347    2741/45396            0/46347         2741/45396
root@sonic-10432:/home/admin#
oot@sonic-10432:/home/admin# show pfc counters | head -n 10
Last cached time was 2021-12-02 20:09:21.025206
    Port Rx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0       0       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0       0       0       0       0       0
  Ethernet6       0       0       0       0       0       0       0       0
  Ethernet8       0       0       0       0       0       0       0       0
 Ethernet10       0       0       0       0       0       0       0       0
 Ethernet12       0       0       0       0       0       0       0       0
root@sonic-10432:/home/admin#

When pause frames are sent , PFCWD storm is detected in Ethenet4 and Ethernet6 

root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        2/1       0/56035   14053/70436                0/0         8634/15297
Ethernet6:3   stormed                        2/1       0/46416   11240/60827               0/69         8499/15431
root@sonic-10432:/home/admin#\

root@sonic-10432:/home/admin#  show pfc counters | grep Tx -A10
    Port Tx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0       0       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0       0       0       0       0       0
  Ethernet6       0       0       0       0       0       0       0       0
  Ethernet8       0       0       0       0       0       0       0       0
 Ethernet10       0       0       0       0       0       0       0       0
 Ethernet12       0       0       0       0       0       0       0       0
 Ethernet14       0       0       0       0       0       0       0       0
 Ethernet16       0       0       0       0       0       0       0       0
root@sonic-10432:/home/admin#

When Pause frames are stopped from both the ports , PFC packets are sent out from either of ports randomly. 

root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        2/2       0/56035  14053/371184                0/0        8634/316045
Ethernet6:3  operational                        2/2       0/46416  11240/361565               0/69        8499/316169
root@sonic-10432:/home/admin# show pfc counters | grep Tx -A10
    Port Tx    PFC0    PFC1    PFC2    PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  ------  ------  ------  ------  ------
  Ethernet0       0       0       0       0       0       0       0       0
  Ethernet2       0       0       0       0       0       0       0       0
  Ethernet4       0       0       0       0       0       0       0       0
  Ethernet6       0       0       0  15,437       0       0       0       0
  Ethernet8       0       0       0       0       0       0       0       0
 Ethernet10       0       0       0       0       0       0       0       0
 Ethernet12       0       0       0       0       0       0       0       0
 Ethernet14       0       0       0       0       0       0       0       0
 Ethernet16       0       0       0       0       0       0       0       0
root@sonic-10432:/home/admin# bcmcmd "show c ce3"
show c ce3
CDMIB_T1518.ce3             :               790,227             +12,021          10,002/s
CDMIB_T64.ce3               :                86,988              +4,777           3,975/s
CDMIB_TPFC3.ce3             :                86,973              +4,777           3,975/s
CDMIB_TBYT.ce3              :           821,540,090         +19,389,312       5,636,129/s
CDMIB_TXPP.ce3              :                86,973              +4,777           3,975/s
CDMIB_TXCF.ce3              :                86,973              +4,777           3,975/s
CDMIB_TPOK.ce3              :               790,274             +12,021          10,002/s
CDMIB_TPKT.ce3              :               877,247             +16,798          13,976/s
CDMIB_TUCA.ce3              :               796,687             +18,476           5,371/s
UC_PERQ_PKT(3).ce3          :               800,057             +15,995           1,672/s
UC_PERQ_BYTE(3).ce3         :           819,258,368         +16,378,880       1,711,632/s
drivshell>
root@sonic-10432:/home/admin# bcmcmd "show c ce3"
show c ce3
CDMIB_T1518.ce3             :               826,763             +36,536           9,935/s
CDMIB_T64.ce3               :               101,507             +14,519           3,949/s
CDMIB_TPFC3.ce3             :               101,492             +14,519           3,949/s
CDMIB_TBYT.ce3              :           853,108,090         +31,568,000          43,307/s
CDMIB_TXPP.ce3              :               101,492             +14,519           3,949/s
CDMIB_TXCF.ce3              :               101,492             +14,519           3,949/s
CDMIB_TPOK.ce3              :               826,810             +36,536           9,935/s
CDMIB_TPKT.ce3              :               928,302             +51,055          13,883/s
CDMIB_TUCA.ce3              :               826,768             +30,081              41/s
UC_PERQ_PKT(3).ce3          :               828,659             +28,602           1,315/s
UC_PERQ_BYTE(3).ce3         :           848,546,816         +29,288,448       1,346,945/s
drivshell>

Describe the results you expected:

PFC frames should not get generated and traffic should flow without any drop.

Output of show version:

root@sonic-10432:~# show  ver

SONiC Software Version: SONiC.20201231.44
Distribution: Debian 10.11
Kernel: 4.19.0-12-2-amd64
Build commit: 63ef7f53bc
Build date: Wed Nov 24 02:53:38 UTC 2021
Built by: cloudtest@876d3befc000000

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH0J5HDGCET0094A0009
Uptime: 19:44:12 up  3:43,  1 user,  load average: 0.62, 0.80, 0.94

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-brcm          20201231.44         9f91a09d0e9c        672MB
docker-syncd-brcm          latest              9f91a09d0e9c        672MB
docker-teamd               20201231.44         48053e1d92e8        390MB
docker-teamd               latest              48053e1d92e8        390MB
docker-router-advertiser   20201231.44         b6f6ccc4ba7f        380MB
docker-router-advertiser   latest              b6f6ccc4ba7f        380MB
docker-platform-monitor    20201231.44         76c085c0a328        561MB
docker-platform-monitor    latest              76c085c0a328        561MB
docker-lldp                20201231.44         a4a5ad2cde5e        420MB
docker-lldp                latest              a4a5ad2cde5e        420MB
docker-snmp                20201231.44         2439767a3279        422MB
docker-snmp                latest              2439767a3279        422MB
docker-dhcp-relay          20201231.44         aee1d0cc3dbb        393MB
docker-dhcp-relay          latest              aee1d0cc3dbb        393MB
docker-database            20201231.44         da24c051253b        379MB
docker-database            latest              da24c051253b        379MB
docker-orchagent           20201231.44         820bdc795273        408MB
docker-orchagent           latest              820bdc795273        408MB
docker-sonic-telemetry     20201231.44         178c953e9f05        469MB
docker-sonic-telemetry     latest              178c953e9f05        469MB
docker-mux                 20201231.44         bcd196bd7a3c        432MB
docker-mux                 latest              bcd196bd7a3c        432MB
docker-fpm-frr             20201231.44         0e266910f4b7        408MB
docker-fpm-frr             latest              0e266910f4b7        408MB
docker-sonic-restapi       20201231.44         6b791c7e73e3        345MB
docker-sonic-restapi       latest              6b791c7e73e3        345MB
docker-acms                20201231.44         d0848884cc78        181MB
docker-acms                latest              d0848884cc78        181MB
k8s.gcr.io/pause           3.4.1               0f8457a4c2ec        683kB

root@sonic-10432:~#

Output of show techsupport:

sonic_dump_sonic-10432_20211202_201617.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

chitra-raghavan commented 2 years ago

Broadcom CSP raised by Vineet