search

agherzan / meta-raspberrypi

Yocto/OE BSP layer for the Raspberry Pi boards
https://www.yoctoproject.org/
MIT License
510 stars 398 forks source link

[dunfell] rcu self-detected stall in events usbnet_deferred_kevent on CM3+ #1159

Open Spark1114 opened 1 year ago

Spark1114 commented 1 year ago

Description

Every time I use ModemManager with NetworkManager, I get a rcu self-detected stall on CPU when the NetworkManager starts the connection. Output of uname -a:

Linux rpi-cm3 5.4.72-v7 #1 SMP Mon Oct 19 11:12:20 UTC 2020 armv7l GNU/Linux

I updated the meta-raspberrypi and other yocto layers to the latest dunfell version available.

For the moment I am stuck and I don't know how to keep debugging to find a fix.

Steps to reproduce the issue:

  1. Configure USB modem connection for NetworkManager
  2. Power on the modem
  3. System freeze (100% CPU load on NetworkManager)
  4. RCU message in the console

Describe the results you received:

[ 3035.217228] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3035.222824] rcu:     0-....: (2099 ticks this GP) idle=936/1/0x40000002 softirq=88620/88620 fqs=923
[ 3035.231694]  (t=2100 jiffies g=173237 q=233)
[ 3035.235961] NMI backtrace for cpu 0
[ 3035.239453] CPU: 0 PID: 592 Comm: kworker/0:0 Tainted: G         C        5.4.72-v7 #1
[ 3035.247365] Hardware name: BCM2835
[ 3035.250780] Workqueue: events usbnet_deferred_kevent
[ 3035.255741] Backtrace:
[ 3035.258198] [<8010e244>] (dump_backtrace) from [<8010e5bc>] (show_stack+0x20/0x24)
[ 3035.265769]  r7:ffffffff r6:00000000 r5:600b0193 r4:80e9df30
[ 3035.271434] [<8010e59c>] (show_stack) from [<80980a90>] (dump_stack+0xd4/0x118)
[ 3035.278746] [<809809bc>] (dump_stack) from [<80987d90>] (nmi_cpu_backtrace+0x98/0xcc)
[ 3035.286579]  r10:80e128c0 r9:80e128c0 r8:800b0193 r7:00000000 r6:00000000 r5:00000000
[ 3035.294406]  r4:00000000 r3:00000000
[ 3035.297984] [<80987cf8>] (nmi_cpu_backtrace) from [<80987ebc>] (nmi_trigger_cpumask_backtrace+0xf8/0x140)
[ 3035.307548]  r5:80e0936c r4:8010fbe8
[ 3035.311129] [<80987dc4>] (nmi_trigger_cpumask_backtrace) from [<801115f8>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[ 3035.321738]  r7:80a01fe0 r6:80e05fac r5:80e06048 r4:00000000
[ 3035.327405] [<801115d8>] (arch_trigger_cpumask_backtrace) from [<801a2a28>] (rcu_dump_cpu_stacks+0xb4/0xdc)
[ 3035.337152] [<801a2974>] (rcu_dump_cpu_stacks) from [<801a18ac>] (rcu_sched_clock_irq+0x800/0x9f8)
[ 3035.346115]  r10:80e04d00 r9:39d24000 r8:80edcb30 r7:80e06038 r6:80da90c0 r5:baacd0c0
[ 3035.353942]  r4:80e128c0 r3:80e128c0
[ 3035.357525] [<801a10ac>] (rcu_sched_clock_irq) from [<801abe5c>] (update_process_times+0x40/0xac)
[ 3035.366400]  r10:b901e000 r9:801c05bc r8:00000000 r7:b901fdc8 r6:b09d35b5 r5:ffffe000
[ 3035.374226]  r4:00000000
[ 3035.376766] [<801abe1c>] (update_process_times) from [<801c037c>] (tick_sched_handle+0x64/0x70)
[ 3035.385462]  r5:000002c2 r4:baac6888
[ 3035.389040] [<801c0318>] (tick_sched_handle) from [<801c0618>] (tick_sched_timer+0x5c/0xb8)
[ 3035.397397] [<801c05bc>] (tick_sched_timer) from [<801acec0>] (__hrtimer_run_queues+0x1a4/0x384)
[ 3035.406182]  r7:baac6618 r6:baac6540 r5:baac6580 r4:baac6888
[ 3035.411848] [<801acd1c>] (__hrtimer_run_queues) from [<801adcf8>] (hrtimer_interrupt+0x130/0x2d0)
[ 3035.420723]  r10:ffffffff r9:baac65c8 r8:baac65f0 r7:baac6618 r6:00000003 r5:200b0193
[ 3035.428549]  r4:baac6540
[ 3035.431088] [<801adbc8>] (hrtimer_interrupt) from [<807d9034>] (arch_timer_handler_phys+0x38/0x40)
[ 3035.440050]  r10:00000000 r9:b901e000 r8:ba464000 r7:000000a2 r6:80e6a40c r5:ba472080
[ 3035.447876]  r4:ba471d00
[ 3035.450415] [<807d8ffc>] (arch_timer_handler_phys) from [<80193c50>] (handle_percpu_devid_irq+0x8c/0x240)
[ 3035.459987] [<80193bc4>] (handle_percpu_devid_irq) from [<8018cfb4>] (generic_handle_irq+0x34/0x44)
[ 3035.469035]  r10:00000000 r9:b901e000 r8:ba464000 r7:00000000 r6:00000001 r5:00000000
[ 3035.476862]  r4:80da8464 r3:80193bc4
[ 3035.480441] [<8018cf80>] (generic_handle_irq) from [<8018d718>] (__handle_domain_irq+0x6c/0xc0)
[ 3035.489143] [<8018d6ac>] (__handle_domain_irq) from [<80102224>] (bcm2836_arm_irqchip_handle_irq+0x64/0xa4)
[ 3035.498887]  r9:b901e000 r8:b9e10400 r7:b901fdfc r6:ffffffff r5:600b0013 r4:8076be50
[ 3035.506633] [<801021c0>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[ 3035.515501] Exception stack(0xb901fdc8 to 0xb901fe10)
[ 3035.520554] fdc0:                   00000000 b9074a80 80e05f88 80e05f88 b9074a80 00000000
[ 3035.528735] fde0: ffffff98 b9e40000 b9e10400 b9074a80 00000000 b901fe44 00000003 b901fe18
[ 3035.536911] fe00: 00000000 8076be50 600b0013 ffffffff
[ 3035.541968] [<8076bd94>] (dwc_otg_urb_dequeue) from [<8073b2ac>] (unlink1+0x3c/0x11c)
[ 3035.549800]  r8:00000006 r7:ffffff98 r6:ffffff98 r5:b9e10400 r4:b9074a80
[ 3035.556504] [<8073b270>] (unlink1) from [<8073ce0c>] (usb_hcd_unlink_urb+0x78/0x94)
[ 3035.564164]  r9:b9074a80 r8:00000006 r7:b7fe3c00 r6:ffffff98 r5:a00b0013 r4:b9074a80
[ 3035.571911] [<8073cd94>] (usb_hcd_unlink_urb) from [<8073e298>] (usb_unlink_urb+0x3c/0x58)
[ 3035.580175]  r7:0007fcf2 r6:b8b36dfc r5:600b0013 r4:b8b36df0
[ 3035.585839] [<8073e25c>] (usb_unlink_urb) from [<8072eb18>] (unlink_urbs.constprop.0+0x8c/0xc4)
[ 3035.594543] [<8072ea8c>] (unlink_urbs.constprop.0) from [<8072f158>] (__handle_link_change.part.0+0x30/0x68)
[ 3035.604373]  r9:00000000 r8:00000000 r7:baacf700 r6:b8b36d40 r5:b7f22880 r4:b8b36d40
[ 3035.612122] [<8072f128>] (__handle_link_change.part.0) from [<8072ffbc>] (usbnet_deferred_kevent+0xe8/0x32c)
[ 3035.621947]  r5:b7f22880 r4:b8b36e84
[ 3035.625529] [<8072fed4>] (usbnet_deferred_kevent) from [<8013ede8>] (process_one_work+0x214/0x538)
[ 3035.634488]  r7:baacf700 r6:baacc4c0 r5:b7f22880 r4:b8b36e84
[ 3035.640153] [<8013ebd4>] (process_one_work) from [<8013f158>] (worker_thread+0x4c/0x580)
[ 3035.648247]  r10:baacc4c0 r9:80e04d00 r8:baacc4d8 r7:00000008 r6:baacc4c0 r5:b7f22894
[ 3035.656072]  r4:b7f22880
[ 3035.658610] [<8013f10c>] (worker_thread) from [<80146684>] (kthread+0x144/0x170)
[ 3035.666010]  r10:b7f93e7c r9:b953539c r8:b7f22880 r7:b901e000 r6:00000000 r5:b9536180
[ 3035.673835]  r4:b9535380
[ 3035.676371] [<80146540>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[ 3035.683589] Exception stack(0xb901ffb0 to 0xb901fff8)
[ 3035.688640] ffa0:                                     00000000 00000000 00000000 00000000
[ 3035.696819] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3035.704998] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 3035.711615]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80146540
[ 3035.719440]  r4:b9536180
[ 3098.247229] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3098.252824] rcu:     0-....: (8353 ticks this GP) idle=936/1/0x40000002 softirq=88620/88620 fqs=3589
[ 3098.261782]  (t=8403 jiffies g=173237 q=904)
[ 3098.266050] NMI backtrace for cpu 0
[ 3098.269541] CPU: 0 PID: 592 Comm: kworker/0:0 Tainted: G         C        5.4.72-v7 #1
[ 3098.277454] Hardware name: BCM2835
[ 3098.280869] Workqueue: events usbnet_deferred_kevent
[ 3098.285831] Backtrace:
[ 3098.288288] [<8010e244>] (dump_backtrace) from [<8010e5bc>] (show_stack+0x20/0x24)
[ 3098.295860]  r7:ffffffff r6:00000000 r5:600b0193 r4:80e9df30
[ 3098.301524] [<8010e59c>] (show_stack) from [<80980a90>] (dump_stack+0xd4/0x118)
[ 3098.308837] [<809809bc>] (dump_stack) from [<80987d90>] (nmi_cpu_backtrace+0x98/0xcc)
[ 3098.316670]  r10:80e128c0 r9:80e128c0 r8:800b0193 r7:00000000 r6:00000000 r5:00000000
[ 3098.324497]  r4:00000000 r3:00000000
[ 3098.328075] [<80987cf8>] (nmi_cpu_backtrace) from [<80987ebc>] (nmi_trigger_cpumask_backtrace+0xf8/0x140)
[ 3098.337639]  r5:80e0936c r4:8010fbe8
[ 3098.341219] [<80987dc4>] (nmi_trigger_cpumask_backtrace) from [<801115f8>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[ 3098.351828]  r7:80a01fe0 r6:80e05fac r5:80e06048 r4:00000000
[ 3098.357496] [<801115d8>] (arch_trigger_cpumask_backtrace) from [<801a2a28>] (rcu_dump_cpu_stacks+0xb4/0xdc)
[ 3098.367243] [<801a2974>] (rcu_dump_cpu_stacks) from [<801a18ac>] (rcu_sched_clock_irq+0x800/0x9f8)
[ 3098.376205]  r10:80e04d00 r9:39d24000 r8:80edcb30 r7:80e06038 r6:80da90c0 r5:baacd0c0
[ 3098.384032]  r4:80e128c0 r3:80e128c0
[ 3098.387615] [<801a10ac>] (rcu_sched_clock_irq) from [<801abe5c>] (update_process_times+0x40/0xac)
[ 3098.396491]  r10:b901e000 r9:801c05bc r8:00000000 r7:b901fe00 r6:5d7eaee9 r5:ffffe000
[ 3098.404316]  r4:00000000
[ 3098.406856] [<801abe1c>] (update_process_times) from [<801c037c>] (tick_sched_handle+0x64/0x70)
[ 3098.415551]  r5:000002d1 r4:baac6888
[ 3098.419130] [<801c0318>] (tick_sched_handle) from [<801c0618>] (tick_sched_timer+0x5c/0xb8)
[ 3098.427486] [<801c05bc>] (tick_sched_timer) from [<801acec0>] (__hrtimer_run_queues+0x1a4/0x384)
[ 3098.436272]  r7:baac6618 r6:baac6540 r5:baac6580 r4:baac6888
[ 3098.441938] [<801acd1c>] (__hrtimer_run_queues) from [<801adcf8>] (hrtimer_interrupt+0x130/0x2d0)
[ 3098.450813]  r10:ffffffff r9:baac65c8 r8:baac65f0 r7:baac6618 r6:00000003 r5:200b0193
[ 3098.458638]  r4:baac6540
[ 3098.461178] [<801adbc8>] (hrtimer_interrupt) from [<807d9034>] (arch_timer_handler_phys+0x38/0x40)
[ 3098.470140]  r10:00000000 r9:b901e000 r8:ba464000 r7:000000a2 r6:80e6a40c r5:ba472080
[ 3098.477966]  r4:ba471d00
[ 3098.480505] [<807d8ffc>] (arch_timer_handler_phys) from [<80193c50>] (handle_percpu_devid_irq+0x8c/0x240)
[ 3098.490077] [<80193bc4>] (handle_percpu_devid_irq) from [<8018cfb4>] (generic_handle_irq+0x34/0x44)
[ 3098.499126]  r10:00000000 r9:b901e000 r8:ba464000 r7:00000000 r6:00000001 r5:00000000
[ 3098.506953]  r4:80da8464 r3:80193bc4
[ 3098.510531] [<8018cf80>] (generic_handle_irq) from [<8018d718>] (__handle_domain_irq+0x6c/0xc0)
[ 3098.519234] [<8018d6ac>] (__handle_domain_irq) from [<80102224>] (bcm2836_arm_irqchip_handle_irq+0x64/0xa4)
[ 3098.528978]  r9:b901e000 r8:00000006 r7:b901fe34 r6:ffffffff r5:600b0013 r4:806b46f8
[ 3098.536724] [<801021c0>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[ 3098.545593] Exception stack(0xb901fe00 to 0xb901fe48)
[ 3098.550648] fe00: 00000000 b7fe3c94 0000001d 0000001e ffffff8d a00b0013 ffffff98 b7fe3c00
[ 3098.558828] fe20: 00000006 b8407200 00000000 b901fe5c 0000001e b901fe50 00000000 806b46f8
[ 3098.567002] fe40: 600b0013 ffffffff
[ 3098.570497] [<806b46d8>] (put_device) from [<807321dc>] (usb_put_dev+0x24/0x28)
[ 3098.577811] [<807321b8>] (usb_put_dev) from [<8073ce1c>] (usb_hcd_unlink_urb+0x88/0x94)
[ 3098.585818] [<8073cd94>] (usb_hcd_unlink_urb) from [<8073e298>] (usb_unlink_urb+0x3c/0x58)
[ 3098.594083]  r7:001fa3b3 r6:b8b36dfc r5:600b0013 r4:b8b36df0
[ 3098.599746] [<8073e25c>] (usb_unlink_urb) from [<8072eb18>] (unlink_urbs.constprop.0+0x8c/0xc4)
[ 3098.608450] [<8072ea8c>] (unlink_urbs.constprop.0) from [<8072f158>] (__handle_link_change.part.0+0x30/0x68)
[ 3098.618280]  r9:00000000 r8:00000000 r7:baacf700 r6:b8b36d40 r5:b7f22880 r4:b8b36d40
[ 3098.626029] [<8072f128>] (__handle_link_change.part.0) from [<8072ffbc>] (usbnet_deferred_kevent+0xe8/0x32c)
[ 3098.635853]  r5:b7f22880 r4:b8b36e84
[ 3098.639436] [<8072fed4>] (usbnet_deferred_kevent) from [<8013ede8>] (process_one_work+0x214/0x538)
[ 3098.648395]  r7:baacf700 r6:baacc4c0 r5:b7f22880 r4:b8b36e84
[ 3098.654060] [<8013ebd4>] (process_one_work) from [<8013f158>] (worker_thread+0x4c/0x580)
[ 3098.662153]  r10:baacc4c0 r9:80e04d00 r8:baacc4d8 r7:00000008 r6:baacc4c0 r5:b7f22894
[ 3098.669979]  r4:b7f22880
[ 3098.672517] [<8013f10c>] (worker_thread) from [<80146684>] (kthread+0x144/0x170)
[ 3098.679917]  r10:b7f93e7c r9:b953539c r8:b7f22880 r7:b901e000 r6:00000000 r5:b9536180
[ 3098.687742]  r4:b9535380
[ 3098.690278] [<80146540>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[ 3098.697497] Exception stack(0xb901ffb0 to 0xb901fff8)
[ 3098.702547] ffa0:                                     00000000 00000000 00000000 00000000
[ 3098.710727] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3098.718904] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 3098.725521]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80146540
[ 3098.733347]  r4:b9536180

Describe the results you expected:

No stall and the system working properly

Additional information you deem important (e.g. issue happens only occasionally):

I tried the same modem and configuration on my Ubuntu 5.15.0-67-generic desktop and everything is working properly.

Additional details (revisions used, host distro, etc.):

-

Thanks for your help !