Open mdrodrigo opened 2 weeks ago
It looks as though something is corrupting kernel memory - some memory locations that are being interpreted as addresses actually seem to hold text strings. User space code should not be able to write to kernel memory in this way, so I suspect you have a bad driver.
When you say CDC Port, are you referring to using the dwc2 driver to bring up the USB interface on the power port as a device running a virtual serial interface?
For more useful diagnostic output, can you boot with slub_debug=FPUZ
in /boot/firmware/config.txt and provoke a crash?
Hi @P33M
For more useful diagnostic output, can you boot with
slub_debug=FPUZ
in /boot/firmware/config.txt and provoke a crash?
The output with slub_debug=FPUZ
[ 88.504658] 8<--- cut here ---
[ 88.507791] Unable to handle kernel paging request at virtual address 0a0d3458 when read
[ 88.516006] [0a0d3458] *pgd=00000000
[ 88.519655] Internal error: Oops: 5 [#1] SMP ARM
[ 88.524345] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep vc4 brcmfmac_wcc snd_soc_hdmi_codec drm_display_helper cec hci_uart drm_dma_helper btbcm drm_kms_helper brcmfmac bluetooth brcmutil snd_soc_core bcm2835_isp(C) bcm2835_v4l2(C) cfg80211 binfmt_misc bcm2835_codec(C) cdc_acm v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_dma_contig raspberrypi_hwmon snd_compress videobuf2_memops snd_pcm_dmaengine videobuf2_v4l2 videodev i2c_bcm2835 snd_bcm2835(C) snd_pcm ecdh_generic ecc rfkill videobuf2_common snd_timer mc snd raspberrypi_gpiomem vc_sm_cma(C) fixed uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6
[ 88.594261] CPU: 2 PID: 1498 Comm: python3 Tainted: G C 6.6.56-v7+ #1804
[ 88.602487] Hardware name: BCM2835
[ 88.605940] PC is at __kmem_cache_alloc_node+0x110/0x48c
[ 88.611350] LR is at 0xb5c7
[ 88.614186] pc : [<8036041c>] lr : [<0000b5c7>] psr: a0000093
[ 88.620544] sp : bbf41cd0 ip : bbf41cd0 fp : aece8300
[ 88.625846] r10: 00000000 r9 : 0000002c r8 : 811057c4
[ 88.631148] r7 : 00000920 r6 : 0a0d3438 r5 : 00000000 r4 : 81401100
[ 88.637774] r3 : 00000020 r2 : 39a59000 r1 : baaf4378 r0 : 0000b5c6
[ 88.644400] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 88.651750] Control: 10c5383d Table: 0285006a DAC: 00000055
[ 88.657613] Register r0 information: non-paged memory
[ 88.662788] Register r1 information: non-slab/vmalloc memory
[ 88.668558] Register r2 information: non-paged memory
[ 88.673687] Register r3 information: non-paged memory
[ 88.678816] Register r4 information: slab kmem_cache start 81401100 pointer offset 0 size 124
[ 88.687493] Register r5 information: NULL pointer
[ 88.692270] Register r6 information: non-paged memory
[ 88.697397] Register r7 information: non-paged memory
[ 88.702524] Register r8 information: non-slab/vmalloc memory
[ 88.708271] Register r9 information: non-paged memory
[ 88.713399] Register r10 information: NULL pointer
[ 88.718263] Register r11 information: slab kmalloc-128 start aece8300 pointer offset 0 size 128
[ 88.727114] Register r12 information: 2-page vmalloc region starting at 0xbbf40000 allocated at kernel_clone+0x9c/0x374
[ 88.738077] Process python3 (pid: 1498, stack limit = 0xd564c68c)
[ 88.744265] Stack: (0xbbf41cd0 to 0xbbf42000)
[ 88.748689] 1cc0: 84400240 00225510 0000b5c7 ffffffff
[ 88.756991] 1ce0: 00000000 0000002c 80883e18 81401100 aece8320 00000920 816cd800 00000000
[ 88.765294] 1d00: aece8300 80317bc0 80883e18 00000000 aece8300 80317bc0 8087e804 816cd800
[ 88.773596] 1d20: 84b6e140 00000001 aece8320 81749e00 816cd800 00000000 aece8300 80883e18
[ 88.781898] 1d40: 816cd800 84b6e140 00000001 8087d3c8 00000000 bbf41da0 00000000 00000002
[ 88.790200] 1d60: 828b5700 84b6e140 81749e00 8087f014 00000200 00000000 00000000 00000001
[ 88.798502] 1d80: 00000000 81743e10 60000093 9fff39ec 00000002 828b5700 00000000 81749e00
[ 88.806804] 1da0: 84fa8400 00000820 828b570c 821bc8c4 00000000 8084efcc 81525b00 81554c00
[ 88.815106] 1dc0: 81525b05 80b26c40 83450000 80b26c40 83450000 80b26c40 81401300 8085f2b4
[ 88.823409] 1de0: 00000000 80360350 00004706 ffffffff 00007348 ffffffff 80850378 00000200
[ 88.831711] 1e00: 8085f2b4 81401300 01c93b58 00002cc0 812ddfc0 8085f464 00000000 80317bc0
[ 88.840012] 1e20: 8085f2b4 00000002 00000820 9fff39ec 00000001 84947780 84947780 84b6e440
[ 88.848315] 1e40: 84947790 00000000 812ddfc0 821bc8c4 00000000 8085f464 bbf41e70 00000004
[ 88.856617] 1e60: 00000001 80173384 aeca3a20 00000000 00000000 00000001 aece8300 00000000
[ 88.864918] 1e80: 00000001 00000000 828b5700 00000000 84947790 00000000 00000000 825f8180
[ 88.873219] 1ea0: baae9a80 801b728c 825f8180 baae9a40 baae9a80 801b7eac 00000000 00000000
[ 88.881522] 1ec0: 00000000 20000093 00000000 801b7350 00008103 00000000 00000000 01c59fc8
[ 88.889823] 1ee0: 00000200 00000000 00000000 00000000 00000000 00000000 01c59120 9fff39ec
[ 88.898127] 1f00: 00000000 802c550a 00000000 8454af01 01c93b58 00000009 8454af00 aeca3a20
[ 88.906429] 1f20: 01c93b58 8038fe70 00000000 00000008 00000000 00000001 00000161 80100298
[ 88.914731] 1f40: 83450000 00000161 00000000 803d79b8 00000000 00000000 00000000 ae695301
[ 88.923032] 1f60: 00000058 00000000 23d47858 80100298 00000000 00000000 00000000 9fff39ec
[ 88.931333] 1f80: 00000000 00000001 01c59120 00000001 00000036 80100298 83450000 00000036
[ 88.939637] 1fa0: 01c93b58 80100040 00000001 01c59120 00000009 802c550a 01c93b58 7ec6bbb0
[ 88.947940] 1fc0: 00000001 01c59120 00000001 00000036 00004000 00000000 00000001 01c93b58
[ 88.956243] 1fe0: 00000000 7ec6bb98 76e1c000 76d9f0e4 60000010 00000009 00000000 00000000
[ 88.964548] __kmem_cache_alloc_node from __kmalloc+0x4c/0x180
[ 88.970487] __kmalloc from dwc_otg_hcd_qtd_create+0x24/0x88
[ 88.976249] dwc_otg_hcd_qtd_create from dwc_otg_hcd_urb_enqueue+0x64/0x1f8
[ 88.983328] dwc_otg_hcd_urb_enqueue from dwc_otg_urb_enqueue+0x1a4/0x334
[ 88.990226] dwc_otg_urb_enqueue from usb_hcd_submit_urb+0xec/0x950
[ 88.996596] usb_hcd_submit_urb from usbdev_ioctl+0x21ac/0x263c
[ 89.002615] usbdev_ioctl from sys_ioctl+0x110/0xa20
[ 89.007670] sys_ioctl from ret_fast_syscall+0x0/0x4c
[ 89.012808] Exception stack(0xbbf41fa8 to 0xbbf41ff0)
[ 89.017939] 1fa0: 00000001 01c59120 00000009 802c550a 01c93b58 7ec6bbb0
[ 89.026241] 1fc0: 00000001 01c59120 00000001 00000036 00004000 00000000 00000001 01c93b58
[ 89.034541] 1fe0: 00000000 7ec6bb98 76e1c000 76d9f0e4
[ 89.039675] Code: e3560000 13530000 0a0000c6 e594301c (e796a003)
[ 89.045862] ---[ end trace 0000000000000000 ]---
[ 89.050546] note: python3[1498] exited with irqs disabled
[ 89.523524] ------------[ cut here ]------------
[ 89.528236] WARNING: CPU: 3 PID: 58 at drivers/firmware/raspberrypi.c:69 rpi_firmware_property_list+0x200/0x274
[ 89.538514] Firmware transaction timeout
[ 89.538520] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep vc4 brcmfmac_wcc snd_soc_hdmi_codec drm_display_helper cec hci_uart drm_dma_helper btbcm drm_kms_helper brcmfmac bluetooth brcmutil snd_soc_core bcm2835_isp(C) bcm2835_v4l2(C) cfg80211 binfmt_misc bcm2835_codec(C) cdc_acm v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_dma_contig raspberrypi_hwmon snd_compress videobuf2_memops snd_pcm_dmaengine videobuf2_v4l2 videodev i2c_bcm2835 snd_bcm2835(C) snd_pcm ecdh_generic ecc rfkill videobuf2_common snd_timer mc snd raspberrypi_gpiomem vc_sm_cma(C) fixed uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6
[ 89.612422] CPU: 3 PID: 58 Comm: kworker/3:1 Tainted: G D C 6.6.56-v7+ #1804
[ 89.620814] Hardware name: BCM2835
[ 89.624265] Workqueue: events dbs_work_handler
[ 89.628792] unwind_backtrace from show_stack+0x18/0x1c
[ 89.634113] show_stack from dump_stack_lvl+0x50/0x68
[ 89.639253] dump_stack_lvl from __warn+0x84/0x11c
[ 89.644130] __warn from warn_slowpath_fmt+0x11c/0x190
[ 89.649356] warn_slowpath_fmt from rpi_firmware_property_list+0x200/0x274
[ 89.656345] rpi_firmware_property_list from rpi_firmware_property+0x68/0x110
[ 89.663595] rpi_firmware_property from raspberrypi_fw_set_rate+0x54/0xcc
[ 89.670499] raspberrypi_fw_set_rate from clk_change_rate+0xe8/0x450
[ 89.676964] clk_change_rate from clk_core_set_rate_nolock+0x15c/0x2d8
[ 89.683607] clk_core_set_rate_nolock from clk_set_rate+0x38/0x15c
[ 89.689895] clk_set_rate from _opp_config_clk_single+0x2c/0x8c
[ 89.695918] _opp_config_clk_single from _set_opp+0x114/0x3e8
[ 89.701763] _set_opp from dev_pm_opp_set_rate+0x128/0x238
[ 89.707345] dev_pm_opp_set_rate from __cpufreq_driver_target+0x1ac/0x258
[ 89.714254] __cpufreq_driver_target from od_dbs_update+0xc4/0x17c
[ 89.720541] od_dbs_update from dbs_work_handler+0x34/0x60
[ 89.726116] dbs_work_handler from process_one_work+0x160/0x378
[ 89.732141] process_one_work from worker_thread+0x2a4/0x4f0
[ 89.737899] worker_thread from kthread+0xe8/0x104
[ 89.742778] kthread from ret_from_fork+0x14/0x38
[ 89.747562] Exception stack(0xbbdc9fb0 to 0xbbdc9ff8)
[ 89.752690] 9fa0: 00000000 00000000 00000000 00000000
[ 89.760993] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 89.769293] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 89.776029] ---[ end trace 0000000000000000 ]---
[ 89.780742] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -110
[ 90.793537] hwmon hwmon1: Failed to get throttled (-110)
[ 91.833537] raspberrypi-exp-gpio soc:firmware:expgpio: Failed to get GPIO 4 state (-110 0)
[ 92.873525] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 93.923521] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 94.953521] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 94.959269] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -110
[ 96.003550] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 97.043525] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 108.403544] raspberrypi-exp-gpio soc:firmware:expgpio: Failed to get GPIO 4 state (-110 84)
[ 109.443541] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 109.513532] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 109.519536] rcu: 0-...0: (0 ticks this GP) idle=3904/1/0x40000000 softirq=5999/5999 fqs=1000
[ 109.528189] rcu: (detected by 3, t=2103 jiffies, g=4541, q=640 ncpus=4)
[ 109.534986] Sending NMI from CPU 3 to CPUs 0:
[ 110.483540] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 111.523541] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 112.563542] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 113.603543] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 119.539404] rcu: rcu_sched kthread timer wakeup didn't happen for 1002 jiffies! g4541 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[ 119.550694] rcu: Possible timer handling issue on cpu=3 timer-softirq=1881
[ 119.557753] rcu: rcu_sched kthread starved for 1005 jiffies! g4541 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
[ 119.568073] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[ 119.577155] rcu: RCU grace-period kthread stack dump:
[ 119.582273] task:rcu_sched state:I stack:0 pid:16 ppid:2 flags:0x00000000
[ 119.590751] __schedule from schedule+0x54/0xe8
[ 119.595361] schedule from schedule_timeout+0x84/0x158
[ 119.600585] schedule_timeout from rcu_gp_fqs_loop+0x114/0x45c
[ 119.606514] rcu_gp_fqs_loop from rcu_gp_kthread+0x12c/0x16c
[ 119.612257] rcu_gp_kthread from kthread+0xe8/0x104
[ 119.617211] kthread from ret_from_fork+0x14/0x38
[ 119.621985] Exception stack(0xbb869fb0 to 0xbb869ff8)
[ 119.627106] 9fa0: 00000000 00000000 00000000 00000000
[ 119.635399] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 119.643691] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 119.650398] rcu: Stack dump where RCU GP kthread last ran:
[ 119.655957] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D WC 6.6.56-v7+ #1804
[ 119.664076] Hardware name: BCM2835
[ 119.667519] PC is at default_idle_call+0x38/0xa0
[ 119.672203] LR is at ct_kernel_enter.constprop.0+0x44/0xa4
[ 119.677767] pc : [<80b24d60>] lr : [<80b24908>] psr: 60000013
[ 119.684119] sp : bb879fb8 ip : 000054e2 fp : 00000000
[ 119.689413] r10: 00000000 r9 : 80e28bd4 r8 : 81256d50
[ 119.694708] r7 : 81105078 r6 : 815ad8c0 r5 : 8110501c r4 : 815ad8c0
[ 119.701326] r3 : 60000093 r2 : 00000003 r1 : 00000001 r0 : 00000003
[ 119.707943] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 119.715179] Control: 10c5383d Table: 0465006a DAC: 00000055
[ 119.721001] default_idle_call from do_idle+0xbc/0x11c
[ 119.726220] do_idle from cpu_startup_entry+0x30/0x34
[ 119.731348] cpu_startup_entry from secondary_start_kernel+0x11c/0x124
[ 119.737976] secondary_start_kernel from 0x101280
[ 125.043556] raspberrypi-exp-gpio soc:firmware:expgpio: Failed to get GPIO 4 state (-110 84)
[ 126.083552] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 127.123549] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 128.163550] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 129.193552] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
[ 130.243549] i2c-bcm2835 3f805000.i2c: i2c transfer timed out
Hi @pelwell
It looks as though something is corrupting kernel memory - some memory locations that are being interpreted as addresses actually seem to hold text strings. User space code should not be able to write to kernel memory in this way, so I suspect you have a bad driver.
When you say CDC Port, are you referring to using the dwc2 driver to bring up the USB interface on the power port as a device running a virtual serial interface?
The raspberrypi is in Host mode, as the other device is providing a device (using a g_serial).
Describe the bug
We have a device that uses CDC Port for communicating and when we send data to the device, we receive this set of kernel Oops.
Steps to reproduce the behaviour
Any communication with this specific device, triggers the Oops.
Device (s)
Raspberry Pi 3 Mod. B+, Raspberry Pi 4 Mod. B
System
Raspberry Pi reference 2024-07-04 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 0b115f302a8f1e5bd3523614d7f45b9d447434c7, stage4
Oct 10 2024 18:27:31 Copyright (c) 2012 Broadcom version 0e63dd5ec215290186b4f3a41496b4425155878e (clean) (release) (start)
Linux raspberrypi 6.6.56-v7+ #1804 SMP Thu Oct 10 18:20:27 BST 2024 armv7l GNU/Linux
Logs
Additional context
No response