Open nazeerhussainf opened 3 years ago
is this a typo?
No. That msg should only get printed if PFC frames are received on lossy queues. Need to understand why it is showing up for 3 and 4
This message gets printed for unset bits of the pfcMask associated with a port. Since 3 and 4 are lossless queues, bits 3 and 4 are set by default in the port pfcMask. During a pfc storm on a particular queue, we unset that queue from the pfcMask of the corresponding port which leads to this message getting printed for the queue that is currently stormed.
Description
When pause frames are sent on lossless queues 3 and 4, they are named as lossy queue in syslog. This issue was found while testing the PFCWD feature on 112019 branch. Attached the complete syslog for reference.
Aug 20 10:26:38.175418 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: Receive notification, storm Aug 20 10:26:38.175607 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: PFC Watchdog detected PFC storm on port Ethernet0, queue index 3, queue id 0x150000000002bb and port id 0x100000000000e. Aug 20 10:28:43.590517 sonic WARNING swss#orchagent: :- pfcFrameCounterCheck: Got PFC 933416836 frame(s) on lossy queue 3 port Ethernet0 Aug 20 10:28:43.621154 sonic WARNING swss#orchagent: :- checkCrmThresholds: ACL_TABLE THRESHOLD_EXCEEDED for TH_PERCENTAGE 100% Used count 2 free count 0 Aug 20 10:28:43.621154 sonic WARNING swss#orchagent: :- checkCrmThresholds: ACL_TABLE THRESHOLD_CLEAR for TH_PERCENTAGE 0% Used count 0 free count 0 Aug 20 10:29:50.539761 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: Receive notification, restore Aug 20 10:29:50.539908 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: PFC Watchdog storm restored on port Ethernet0, queue index 3, queue id 0x150000000002bb and port id 0x100000000000e.
Aug 20 13:08:43.583610 sonic WARNING swss#orchagent: :- pfcFrameCounterCheck: Got PFC 2232283451 frame(s) on lossy queue 3 port Ethernet4 Aug 20 13:08:43.583704 sonic WARNING swss#orchagent: :- pfcFrameCounterCheck: Got PFC 2232283450 frame(s) on lossy queue 4 port Ethernet4 Aug 20 13:08:43.610866 sonic WARNING swss#orchagent: :- checkCrmThresholds: ACL_TABLE THRESHOLD_EXCEEDED for TH_PERCENTAGE 100% Used count 2 free count 0 Aug 20 13:08:43.610952 sonic WARNING swss#orchagent: :- checkCrmThresholds: ACL_TABLE THRESHOLD_CLEAR for TH_PERCENTAGE 0% Used count 0 free count 0 Aug 20 13:09:12.928831 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: Receive notification, restore Aug 20 13:09:12.928984 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: PFC Watchdog storm restored on port Ethernet4, queue index 4, queue id 0x1500000000002e4 and port id 0x100000000000f. Aug 20 13:09:12.932012 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: Receive notification, restore Aug 20 13:09:12.932099 sonic NOTICE swss#orchagent: :- startWdActionOnQueue: PFC Watchdog storm restored on port Ethernet4, queue index 3, queue id 0x1500000000002e3 and port id 0x100000000000f.
config_db.txt syslog.txt
Steps to reproduce the issue:
Aug 20 13:08:43.583610 sonic WARNING swss#orchagent: :- pfcFrameCounterCheck: Got PFC 2232283451 frame(s) on lossy queue 3 port Ethernet4 Aug 20 13:08:43.583704 sonic WARNING swss#orchagent: :- pfcFrameCounterCheck: Got PFC 2232283450 frame(s) on lossy queue 4 port Ethernet4
Describe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
root@sonic:~# show version
SONiC Software Version: SONiC.201911.40-c1feae8a Distribution: Debian 9.13 Kernel: 4.9.0-11-2-amd64 Build commit: c1feae8a Build date: Fri Nov 13 03:56:18 UTC 2020 Built by: johnar@jenkins-worker-7
Platform: x86_64-dellemc_z9264f_c3538-r0 HwSKU: DellEMC-Z9264f-C64 ASIC: broadcom Serial Number: TW0WCXFVDNT009AE0051 Uptime: 19:39:29 up 1:44, 1 user, load average: 0.42, 0.77, 0.72
Docker images: REPOSITORY TAG IMAGE ID SIZE docker-syncd-brcm 201911.40-c1feae8a 51e3f6ebf231 437MB docker-syncd-brcm latest 51e3f6ebf231 437MB docker-platform-monitor 201911.40-c1feae8a aef406b07757 358MB docker-platform-monitor latest aef406b07757 358MB docker-router-advertiser 201911.40-c1feae8a 1e239abb7464 289MB docker-router-advertiser latest 1e239abb7464 289MB docker-fpm-frr 201911.40-c1feae8a 85ba3aef78d9 335MB docker-fpm-frr latest 85ba3aef78d9 335MB docker-sflow 201911.40-c1feae8a f948810b455e 315MB docker-sflow latest f948810b455e 315MB docker-lldp-sv2 201911.40-c1feae8a 9d45ccfe1cc1 312MB docker-lldp-sv2 latest 9d45ccfe1cc1 312MB docker-orchagent 201911.40-c1feae8a 09bf251c8ef5 333MB docker-orchagent latest 09bf251c8ef5 333MB docker-dhcp-relay 201911.40-c1feae8a ac6cd25edf0c 299MB docker-dhcp-relay latest ac6cd25edf0c 299MB docker-database 201911.40-c1feae8a 7f9717c0ecc5 289MB docker-database latest 7f9717c0ecc5 289MB docker-snmp-sv2 201911.40-c1feae8a d47eeff40cc1 348MB docker-snmp-sv2 latest d47eeff40cc1 348MB docker-teamd 201911.40-c1feae8a d367608cdf2c 315MB docker-teamd latest d367608cdf2c 315MB docker-nat 201911.40-c1feae8a fa7092a97800 316MB docker-nat latest fa7092a97800 316MB docker-sonic-mgmt-framework 201911.40-c1feae8a f92887b1b580 431MB docker-sonic-mgmt-framework latest f92887b1b580 431MB docker-sonic-telemetry 201911.40-c1feae8a 30c55297de06 353MB docker-sonic-telemetry latest 30c55297de06 353MB
root@sonic:~#
sonic_dump_sonic_20201202_200638.tar.gz