search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
195 stars 715 forks source link

CPU Stall Issue on the DUT during Test Execution #10704

Open mithun2498 opened 11 months ago

mithun2498 commented 11 months ago

Description

I am facing the CPU Stall Issue on the DUT during test execution. The issue seen is frequent during test execution as a groups.

Describe the results you received: dev-msn2700-01 login: admin Password: [46547.660832] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [46547.667453] rcu: 0-...0: (0 ticks this GP) idle=7fe/1/0x4000000000000000 softirq=824556/824556 fqs=3739010 [46610.680830] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [46610.687450] rcu: 0-...0: (0 ticks this GP) idle=7fe/1/0x4000000000000000 softirq=824556/824556 fqs=3745347 [46673.700829] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [46673.707448] rcu: 0-...0: (0 ticks this GP) idle=7fe/1/0x4000000000000000 softirq=824556/824556 fqs=3752305

Describe the results you expected: Expecting to resolve the stall issue seen.

Additional information you deem important: CPLD Version: root@(none):/# cpldutil -ver CPLD#1 Version: 10 (0x0A) CPLD#2 Version: 14 (0x0E) CPLD#3 Version: 14 (0x0E) FAN CPLD Version: 08 (0x08) CPU CPLD Version: 12 (0x0C)

**Output of `show version`:**

root@dev-msn2700-01:~# show platform summary Platform: x86_64-accton_as7716_32x-r0 HwSKU: Accton-AS7716-32X ASIC: broadcom ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A

root@dev-msn2700-01:~# show version SONiC Software Version: SONiC.202305.0-34728958a SONiC OS Version: 11 Distribution: Debian 11.8 Kernel: 5.10.0-23-2-amd64 Build commit: 34728958a Build date: Tue Oct 24 09:56:44 UTC 2023 Built by: sonic@sfsintel Platform: x86_64-accton_as7716_32x-r0 HwSKU: Accton-AS7716-32X ASIC: broadcom ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A Uptime: 15:55:51 up 33 min, 1 user, load average: 1.99, 1.84, 1.60 Date: Sun 18 Jun 2023 15:55:51

mithun2498 commented 11 months ago

Hi @yxieca @wangxin , Kindly confirm if the CPU stall issue seen is due to the image or the hardware. I have attached the CPU Stall Logs for reference. The issue is also seen in the below image.

Version Details: SONiC Software Version: SONiC.202211.249702-95f387cdd Distribution: Debian 11.6 Kernel: 5.10.0-18-2-amd64 Build commit: 95f387cdd Build date: Sun Apr 9 14:07:05 UTC 2023 Built by: AzDevOps@vmss-soni000TXR Platform: x86_64-accton_as7716_32x-r0 HwSKU: Accton-AS7716-32X ASIC: broadcom ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A Uptime: 11:44:15 up 4:05, 1 user, load average: 2.65, 2.48, 2.32 Date: Mon 08 Aug 2022 11:44:15

root@dev-msn2700-01~# timed out wai.txt