search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
724 stars 1.38k forks source link

[Z9332 Th3] : When PFC pause frames are sent sequentially from 2 or more ports, pfcwd is detected in only one port. #9432

Open chitra-raghavan opened 2 years ago

chitra-raghavan commented 2 years ago

Description

[Z9332 Th3] : When PFC pause frames are sent sequentially from 2 or more ports, pfcwd is detected in only one port. Device : Z9332 ASIC : Th3 SKU : DellEMC-Z9332f-M-O16C64

Steps to reproduce the issue:

Test Setup :

  1. Connect 2 (or more) IXIA ports to DUT
  2. Configure PFCWD drop action on both the ports
  3. Send bidirectional P3 Data Traffic ( say 10000 Packets/sec)
  4. Start P3 pause frames from one port P1 at time t1
  5. Start P3 pause frames from one port P2 at time t2

Describe the results you received:

  1. PFCWD is not detected in port P2 at time t2.
  2. PFCWD is detected in port P2 only when PFC pause frames are stopped in Port P1 and vice versa,

Before sending pause frames : 

root@sonic-10432:/home/admin# show pfcwd config
Changed polling interval to 200ms
      PORT    ACTION    DETECTION TIME    RESTORATION TIME
----------  --------  ----------------  ------------------
 Ethernet4      drop               200                 200
 Ethernet6      drop               200                 200
Ethernet96      drop               200                 200
root@sonic-10432:/home/admin# show interfaces  counters  | grep U
      IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
  Ethernet2        U        0    0.00 B/s      0.00%         0         0         0        2   36.11 B/s      0.00%         0         0         0
  Ethernet4        U  540,401  10.23 MB/s      0.08%         0         0         0  540,397  10.23 MB/s      0.08%         0         0         0
  Ethernet6        U  540,394  10.23 MB/s      0.08%         4         0         0  540,402  10.23 MB/s      0.08%         0         0         0
Ethernet104        U        2    6.82 B/s      0.00%         0         0         0       58  179.96 B/s      0.00%         0         0         0
root@sonic-10432:/home/admin# show pfcwd stats
  QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-------  --------  -------------------------  ------------  ------------  -----------------  -----------------
root@sonic-10432:/home/admin#

Send Pause frames from IXIA to port Ethernet4

 root@sonic-10432:/home/admin# show pfc counters  | head -n 10
Last cached time was 2021-12-02 20:33:56.843280
    Port Rx    PFC0    PFC1    PFC2     PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  -------  ------  ------  ------  ------
  Ethernet0       0       0       0        0       0       0       0       0
  Ethernet2       0       0       0        0       0       0       0       0
  Ethernet4       0       0       0  133,613       0       0       0       0
  Ethernet6       0       0       0        0       0       0       0       0
  Ethernet8       0       0       0        0       0       0       0       0
 Ethernet10       0       0       0        0       0       0       0       0
 Ethernet12       0       0       0        0       0       0       0       0
root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        1/0      0/168470   1637/167528           0/168470        1637/167528
root@sonic-10432:/home/admin#

Send Pause frames from IXIA to port Ethernet6 : Ethernet6 receives pause frames But PFCWD not triggered, 

root@sonic-10432:/home/admin#  show pfc counters  | head -n 10
Last cached time was 2021-12-02 20:33:56.843280
    Port Rx    PFC0    PFC1    PFC2     PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  -------  ------  ------  ------  ------
  Ethernet0       0       0       0        0       0       0       0       0
  Ethernet2       0       0       0        0       0       0       0       0
  Ethernet4       0       0       0  627,201       0       0       0       0
  Ethernet6       0       0       0  111,601       0       0       0       0
  Ethernet8       0       0       0        0       0       0       0       0
 Ethernet10       0       0       0        0       0       0       0       0
 Ethernet12       0       0       0        0       0       0       0       0
root@sonic-10432:/home/admin#  show pfc counters  | head -n 10
Last cached time was 2021-12-02 20:33:56.843280
    Port Rx    PFC0    PFC1    PFC2     PFC3    PFC4    PFC5    PFC6    PFC7
-----------  ------  ------  ------  -------  ------  ------  ------  ------
  Ethernet0       0       0       0        0       0       0       0       0
  Ethernet2       0       0       0        0       0       0       0       0
  Ethernet4       0       0       0  649,294       0       0       0       0
  Ethernet6       0       0       0  133,695       0       0       0       0
  Ethernet8       0       0       0        0       0       0       0       0
 Ethernet10       0       0       0        0       0       0       0       0
 Ethernet12       0       0       0        0       0       0       0       0
root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  --------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3   stormed                        1/0      0/685790   1637/684661           0/685790        1637/684661
root@sonic-10432:/home/admin#

Stop PFC pause to Ethernet4: Now PFCWD is triggered in Ethernet6

root@sonic-10432:/home/admin# show pfcwd stats
      QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet4:3  operational                        1/1     0/1311393  1637/1310456          0/1311393       1637/1310456
Ethernet6:3      stormed                        1/0       0/90769    6782/89758            0/90769         6782/89758
root@sonic-10432:/home/admin#

Describe the results you expected:

Both ports should get triggered when pause frames are received

Output of show version:

root@sonic-10432:~# show  ver

SONiC Software Version: SONiC.20201231.44
Distribution: Debian 10.11
Kernel: 4.19.0-12-2-amd64
Build commit: 63ef7f53bc
Build date: Wed Nov 24 02:53:38 UTC 2021
Built by: cloudtest@876d3befc000000

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH0J5HDGCET0094A0009
Uptime: 19:44:12 up  3:43,  1 user,  load average: 0.62, 0.80, 0.94

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-brcm          20201231.44         9f91a09d0e9c        672MB
docker-syncd-brcm          latest              9f91a09d0e9c        672MB
docker-teamd               20201231.44         48053e1d92e8        390MB
docker-teamd               latest              48053e1d92e8        390MB
docker-router-advertiser   20201231.44         b6f6ccc4ba7f        380MB
docker-router-advertiser   latest              b6f6ccc4ba7f        380MB
docker-platform-monitor    20201231.44         76c085c0a328        561MB
docker-platform-monitor    latest              76c085c0a328        561MB
docker-lldp                20201231.44         a4a5ad2cde5e        420MB
docker-lldp                latest              a4a5ad2cde5e        420MB
docker-snmp                20201231.44         2439767a3279        422MB
docker-snmp                latest              2439767a3279        422MB
docker-dhcp-relay          20201231.44         aee1d0cc3dbb        393MB
docker-dhcp-relay          latest              aee1d0cc3dbb        393MB
docker-database            20201231.44         da24c051253b        379MB
docker-database            latest              da24c051253b        379MB
docker-orchagent           20201231.44         820bdc795273        408MB
docker-orchagent           latest              820bdc795273        408MB
docker-sonic-telemetry     20201231.44         178c953e9f05        469MB
docker-sonic-telemetry     latest              178c953e9f05        469MB
docker-mux                 20201231.44         bcd196bd7a3c        432MB
docker-mux                 latest              bcd196bd7a3c        432MB
docker-fpm-frr             20201231.44         0e266910f4b7        408MB
docker-fpm-frr             latest              0e266910f4b7        408MB
docker-sonic-restapi       20201231.44         6b791c7e73e3        345MB
docker-sonic-restapi       latest              6b791c7e73e3        345MB
docker-acms                20201231.44         d0848884cc78        181MB
docker-acms                latest              d0848884cc78        181MB
k8s.gcr.io/pause           3.4.1               0f8457a4c2ec        683kB

root@sonic-10432:~#

Output of show techsupport:

sonic_dump_sonic-10432_20211202_204243.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

chitra-raghavan commented 2 years ago

Brcm CSP raised by Vineet

zhangyanzhao commented 2 years ago

@chitra-raghavan please work with Vineet on this issue.