search

raspberrypi / firmware

This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
5.11k stars 1.68k forks source link

Pi freezing up 4 times a day - Workqueue: events_freezable mmc_rescan #1522

Open titiviking opened 3 years ago

titiviking commented 3 years ago

I'm having some issues where my Pi freezes completely. It is crashing 4 times a day !!!!! Only thing I can do is to unplug/replug to reboot it.

The Pi is still responding to ping commands, but all the rest is unresponsive (no SSH either, so I can't look at logs/processes when it actually happens).

Syslog:

Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:

kern.log:

Jan  4 15:09:06 raspberrypi kernel: [ 2240.848751] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.594965] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595133] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595214] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595324] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197103] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197284] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197332] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197451] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Linux version 5.10.3-v7+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1386 SMP Sun Dec 27 14:01:57 GMT 2020

My Pi seems to be fully up-to-date as I've performed following already:

I'm completely desperate in this case... My only option I'm evaluating now is to step away from the Pi and go with a full-fledged server :-(

pelwell commented 3 years ago
  1. Are you using Bluetooth, and if so for what?
  2. What's in your config.txt?
  3. What power supply are you using?
  4. What external equipment is attached?
titiviking commented 3 years ago

  1. Are you using Bluetooth, and if so for what? Yes I am. Used to fetch info from a Bluetooth sensor. Data is displayed in Home Assistant.

  2. What's in your config.txt?

    
root@raspberrypi:~ # cat /boot/config.txt
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

uncomment if you get no picture on HDMI for a default "safe" mode

hdmi_safe=1

uncomment this if your display has a black border of unused pixels visible

and your display can output without overscan

disable_overscan=1

uncomment the following to adjust overscan. Use positive numbers if console

goes off screen, and negative if there is too much border

overscan_left=16

overscan_right=16

overscan_top=16

overscan_bottom=16

uncomment to force a console size. By default it will be display's size minus

overscan.

framebuffer_width=1280

framebuffer_height=720

uncomment if hdmi display is not detected and composite is being output

hdmi_force_hotplug=1

uncomment to force a specific HDMI mode (this will force VGA)

hdmi_group=1

hdmi_mode=1

uncomment to force a HDMI mode rather than DVI. This can make audio work in

DMT (computer monitor) modes

hdmi_drive=2

uncomment to increase signal to HDMI, if you have interference, blanking, or

no display

config_hdmi_boost=4

uncomment for composite PAL

sdtv_mode=2

uncomment to overclock the arm. 700 MHz is the default.

arm_freq=800

Uncomment some or all of these to enable the optional hardware interfaces

dtparam=i2c_arm=on

dtparam=i2s=on

dtparam=spi=on

Uncomment this to enable infrared communication.

dtoverlay=gpio-ir,gpio_pin=17

dtoverlay=gpio-ir-tx,gpio_pin=18

Disable WiFi

dtoverlay=disable-wifi

Additional overlays and parameters are documented /boot/overlays/README

Enable audio (loads snd_bcm2835)

dtparam=audio=on

[pi4]

Enable DRM VC4 V3D driver on top of the dispmanx display stack

dtoverlay=vc4-fkms-v3d max_framebuffers=2

[all]

dtoverlay=vc4-fkms-v3d


3. What power supply are you using?
I'm using the USB output of my NAS or TPLink router.
(Now I'm making a tast with real power plug to see if it offers any stability)

4. What external equipment is attached?
None.

Additional info on current running processes (after 10min of runtime):

root@raspberrypi:~ # ps axo comm | sort | uniq -c | sort -rn | head 21 s6-supervise 12 containerd-shim 10 s6-svscan 8 bash 7 docker-proxy 7 dbus 4 mmal-vchiq 3 udevd 3 sshd 2 systemd



![image](https://user-images.githubusercontent.com/73351833/103550983-f6ae7500-4ea9-11eb-94af-278d9fd6b924.png)
popcornmix commented 3 years ago

I'm using the USB output of my NAS or TPLink router. (Now I'm making a tast with real power plug to see if it offers any stability)

This is likely to be the issue. Not guaranteed to get more than 500mA from a USB port which isn't enough for any model of Pi (which Pi model is it?)

titiviking commented 3 years ago

It's a Raspberry Pi 3B

JamesH65 commented 3 years ago

For power requirements see this page. https://github.com/raspberrypi/documentation/tree/master/faqs

titiviking commented 3 years ago

Issue reoccured now... Pi was properly powered through the provided adapter.

kern.log:

Jan  4 20:11:19 raspberrypi kernel: [ 9764.166094] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:17:03 raspberrypi kernel: [10108.186048] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:24:44 raspberrypi kernel: [10108.186587] Bluetooth: hci0: advertising data len corrected 17 -> 3
Jan  4 20:24:50 raspberrypi kernel: [10565.683087] INFO: task kworker/2:3:12050 blocked for more than 122 seconds.
Jan  4 20:24:52 raspberrypi kernel: [10565.683110]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:24:57 raspberrypi kernel: [10565.683120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:25:02 raspberrypi kernel: [10565.683133] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:25:08 raspberrypi kernel: [10565.683172] Workqueue: events_freezable mmc_rescan
Jan  4 20:25:15 raspberrypi kernel: [10565.683192] Backtrace:
Jan  4 20:25:24 raspberrypi kernel: [10565.683222] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:25:33 raspberrypi kernel: [10565.683242]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:25:42 raspberrypi kernel: [10565.683253]  r4:ffffe000
Jan  4 20:25:47 raspberrypi kernel: [10565.683272] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:25:53 raspberrypi kernel: [10565.683285]  r5:82211218 r4:00000002
Jan  4 20:25:58 raspberrypi kernel: [10565.683302] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:26:00 raspberrypi kernel: [10565.683320]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:26:14 raspberrypi kernel: [10565.683331]  r4:82211800
Jan  4 20:26:24 raspberrypi kernel: [10565.683353] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:26:31 raspberrypi kernel: [10565.683366]  r5:82211000 r4:82211000
Jan  4 20:26:41 raspberrypi kernel: [10565.683386] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:26:49 raspberrypi kernel: [10565.683399]  r5:82211000 r4:8221127c
Jan  4 20:26:55 raspberrypi kernel: [10565.683420] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:27:05 raspberrypi kernel: [10565.683438]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:27:17 raspberrypi kernel: [10565.683457] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:27:26 raspberrypi kernel: [10565.683475]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:27:37 raspberrypi kernel: [10565.683486]  r4:86abf900
Jan  4 20:27:48 raspberrypi kernel: [10565.683506] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:28:05 raspberrypi kernel: [10565.683524]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:28:35 raspberrypi kernel: [10565.683535]  r4:9e4771c0
Jan  4 20:28:56 raspberrypi kernel: [10565.683553] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:29:15 raspberrypi kernel: [10565.683566] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:29:29 raspberrypi kernel: [10565.683580] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:29:43 raspberrypi kernel: [10565.683596] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:29:47 raspberrypi kernel: [10565.683611] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:29:54 raspberrypi kernel: [10565.683629]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:29:57 raspberrypi kernel: [10565.683640]  r4:b6004780
Jan  4 20:29:59 raspberrypi kernel: [10688.564295] INFO: task kworker/2:3:12050 blocked for more than 245 seconds.
Jan  4 20:30:05 raspberrypi kernel: [10688.564311]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:30:13 raspberrypi kernel: [10688.564317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:30:15 raspberrypi kernel: [10688.564324] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:30:21 raspberrypi kernel: [10688.564351] Workqueue: events_freezable mmc_rescan
Jan  4 20:30:26 raspberrypi kernel: [10688.564363] Backtrace:
Jan  4 20:30:31 raspberrypi kernel: [10688.564381] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:30:35 raspberrypi kernel: [10688.564393]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:30:41 raspberrypi kernel: [10688.564400]  r4:ffffe000
Jan  4 20:30:45 raspberrypi kernel: [10688.564411] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:30:49 raspberrypi kernel: [10688.564419]  r5:82211218 r4:00000002
Jan  4 20:30:53 raspberrypi kernel: [10688.564429] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:30:57 raspberrypi kernel: [10688.564439]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:30:59 raspberrypi kernel: [10688.564446]  r4:82211800
Jan  4 20:31:06 raspberrypi kernel: [10688.564459] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:31:16 raspberrypi kernel: [10688.564467]  r5:82211000 r4:82211000
Jan  4 20:31:22 raspberrypi kernel: [10688.564478] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:31:25 raspberrypi kernel: [10688.564486]  r5:82211000 r4:8221127c
Jan  4 20:31:28 raspberrypi kernel: [10688.564499] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:31:29 raspberrypi kernel: [10688.564509]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:31:31 raspberrypi kernel: [10688.564521] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:31:34 raspberrypi kernel: [10688.564531]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:31:37 raspberrypi kernel: [10688.564538]  r4:86abf900
Jan  4 20:31:40 raspberrypi kernel: [10688.564550] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:31:43 raspberrypi kernel: [10688.564561]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:31:46 raspberrypi kernel: [10688.564567]  r4:9e4771c0
Jan  4 20:31:48 raspberrypi kernel: [10688.564578] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:31:51 raspberrypi kernel: [10688.564586] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:31:55 raspberrypi kernel: [10688.564594] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:31:57 raspberrypi kernel: [10688.564604] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:32:00 raspberrypi kernel: [10688.564612] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:32:06 raspberrypi kernel: [10688.564622]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:32:11 raspberrypi kernel: [10688.564629]  r4:b6004780
Jan  4 20:32:14 raspberrypi kernel: [10811.445604] INFO: task kworker/2:3:12050 blocked for more than 368 seconds.
Jan  4 20:32:19 raspberrypi kernel: [10811.445620]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:32:21 raspberrypi kernel: [10811.445626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:32:27 raspberrypi kernel: [10811.445634] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:32:34 raspberrypi kernel: [10811.445660] Workqueue: events_freezable mmc_rescan
Jan  4 20:32:44 raspberrypi kernel: [10811.445672] Backtrace:
Jan  4 20:32:48 raspberrypi kernel: [10811.445690] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:32:52 raspberrypi kernel: [10811.445702]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:32:57 raspberrypi kernel: [10811.445709]  r4:ffffe000
Jan  4 20:33:03 raspberrypi kernel: [10811.445719] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:33:08 raspberrypi kernel: [10811.445727]  r5:82211218 r4:00000002
Jan  4 20:33:11 raspberrypi kernel: [10811.445737] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:33:19 raspberrypi kernel: [10811.445748]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:33:23 raspberrypi kernel: [10811.445755]  r4:82211800
Jan  4 20:33:24 raspberrypi kernel: [10811.445767] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:33:28 raspberrypi kernel: [10811.445775]  r5:82211000 r4:82211000
Jan  4 20:33:31 raspberrypi kernel: [10811.445786] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:33:33 raspberrypi kernel: [10811.445794]  r5:82211000 r4:8221127c
Jan  4 20:33:38 raspberrypi kernel: [10811.445807] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:33:43 raspberrypi kernel: [10811.445817]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:33:49 raspberrypi kernel: [10811.445828] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:33:52 raspberrypi kernel: [10811.445839]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:33:54 raspberrypi kernel: [10811.445845]  r4:86abf900
Jan  4 20:33:56 raspberrypi kernel: [10811.445857] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:33:58 raspberrypi kernel: [10811.445868]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:34:04 raspberrypi kernel: [10811.445875]  r4:9e4771c0
Jan  4 20:34:07 raspberrypi kernel: [10811.445886] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:34:11 raspberrypi kernel: [10811.445893] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:34:20 raspberrypi kernel: [10811.445901] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:34:28 raspberrypi kernel: [10811.445911] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:34:36 raspberrypi kernel: [10811.445920] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:34:46 raspberrypi kernel: [10811.445930]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:34:56 raspberrypi kernel: [10811.445936]  r4:b6004780
Jan  4 20:35:05 raspberrypi kernel: [10934.326923] INFO: task kworker/2:3:12050 blocked for more than 491 seconds.
Jan  4 20:35:16 raspberrypi kernel: [10934.326939]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:35:24 raspberrypi kernel: [10934.326945] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:35:33 raspberrypi kernel: [10934.326953] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:35:45 raspberrypi kernel: [10934.326981] Workqueue: events_freezable mmc_rescan
Jan  4 20:35:55 raspberrypi kernel: [10934.326993] Backtrace:
Jan  4 20:36:01 raspberrypi kernel: [10934.327012] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:36:09 raspberrypi kernel: [10934.327024]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:36:18 raspberrypi kernel: [10934.327031]  r4:ffffe000
Jan  4 20:36:24 raspberrypi kernel: [10934.327041] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:36:31 raspberrypi kernel: [10934.327049]  r5:82211218 r4:00000002
Jan  4 20:36:39 raspberrypi kernel: [10934.327059] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:36:53 raspberrypi kernel: [10934.327070]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:37:00 raspberrypi kernel: [10934.327076]  r4:82211800
Jan  4 20:37:07 raspberrypi kernel: [10934.327088] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:37:16 raspberrypi kernel: [10934.327096]  r5:82211000 r4:82211000
Jan  4 20:37:26 raspberrypi kernel: [10934.327108] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:37:33 raspberrypi kernel: [10934.327116]  r5:82211000 r4:8221127c
Jan  4 20:37:40 raspberrypi kernel: [10934.327128] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:37:46 raspberrypi kernel: [10934.327139]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:37:51 raspberrypi kernel: [10934.327150] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:37:57 raspberrypi kernel: [10934.327161]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:38:04 raspberrypi kernel: [10934.327168]  r4:86abf900
Jan  4 20:38:15 raspberrypi kernel: [10934.327180] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:38:23 raspberrypi kernel: [10934.327190]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:38:31 raspberrypi kernel: [10934.327197]  r4:9e4771c0
Jan  4 20:38:41 raspberrypi kernel: [10934.327208] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:38:50 raspberrypi kernel: [10934.327216] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:38:59 raspberrypi kernel: [10934.327224] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:39:09 raspberrypi kernel: [10934.327234] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:39:21 raspberrypi kernel: [10934.327243] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:39:29 raspberrypi kernel: [10934.327253]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:39:32 raspberrypi kernel: [10934.327259]  r4:b6004780
Jan  4 20:39:37 raspberrypi kernel: [11055.198724] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:39:47 raspberrypi kernel: [11055.198779] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:39:55 raspberrypi kernel: [11055.198864] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:01 raspberrypi kernel: [11055.198949] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:10 raspberrypi kernel: [11055.199036] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:17 raspberrypi kernel: [11055.199101] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:25 raspberrypi kernel: [11057.208319] INFO: task kworker/2:3:12050 blocked for more than 614 seconds.
Jan  4 20:40:32 raspberrypi kernel: [11057.208343]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:40:39 raspberrypi kernel: [11057.208353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:40:48 raspberrypi kernel: [11057.208365] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:40:59 raspberrypi kernel: [11057.208405] Workqueue: events_freezable mmc_rescan
Jan  4 20:41:02 raspberrypi kernel: [11057.208424] Backtrace:
Jan  4 20:41:09 raspberrypi kernel: [11057.208455] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:41:17 raspberrypi kernel: [11057.208474]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:41:27 raspberrypi kernel: [11057.208486]  r4:ffffe000
Jan  4 20:41:31 raspberrypi kernel: [11057.208504] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:41:41 raspberrypi kernel: [11057.208517]  r5:82211218 r4:00000002
Jan  4 20:41:53 raspberrypi kernel: [11057.208535] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:42:00 raspberrypi kernel: [11057.208553]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:42:12 raspberrypi kernel: [11057.208564]  r4:82211800
Jan  4 20:42:22 raspberrypi kernel: [11057.208586] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:42:28 raspberrypi kernel: [11057.208599]  r5:82211000 r4:82211000
Jan  4 20:42:35 raspberrypi kernel: [11057.208618] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:42:44 raspberrypi kernel: [11057.208632]  r5:82211000 r4:8221127c
Jan  4 20:42:56 raspberrypi kernel: [11057.208652] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:43:07 raspberrypi kernel: [11057.208670]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:43:17 raspberrypi kernel: [11057.208690] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:43:26 raspberrypi kernel: [11057.208708]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:43:33 raspberrypi kernel: [11057.208719]  r4:86abf900
Jan  4 20:43:41 raspberrypi kernel: [11057.208738] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:43:48 raspberrypi kernel: [11057.208757]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:43:55 raspberrypi kernel: [11057.208768]  r4:9e4771c0
Jan  4 20:44:03 raspberrypi kernel: [11057.208787] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:44:13 raspberrypi kernel: [11057.208799] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:44:23 raspberrypi kernel: [11057.208813] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:44:27 raspberrypi kernel: [11057.208829] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:44:35 raspberrypi kernel: [11057.208844] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:44:44 raspberrypi kernel: [11057.208861]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:44:53 raspberrypi kernel: [11057.208873]  r4:b6004780
Jan  4 20:45:11 raspberrypi kernel: [11180.089644] INFO: task kworker/2:3:12050 blocked for more than 737 seconds.
Jan  4 20:45:27 raspberrypi kernel: [11180.089667]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:45:36 raspberrypi kernel: [11180.089677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:45:47 raspberrypi kernel: [11180.089689] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:45:56 raspberrypi kernel: [11180.089730] Workqueue: events_freezable mmc_rescan
Jan  4 20:46:08 raspberrypi kernel: [11180.089749] Backtrace:
Jan  4 20:46:19 raspberrypi kernel: [11180.089779] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:46:39 raspberrypi kernel: [11180.089799]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:46:54 raspberrypi kernel: [11180.089810]  r4:ffffe000
Jan  4 20:47:04 raspberrypi kernel: [11180.089829] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:47:22 raspberrypi kernel: [11180.089842]  r5:82211218 r4:00000002
Jan  4 20:47:38 raspberrypi kernel: [11180.089859] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:47:46 raspberrypi kernel: [11180.089877]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:47:51 raspberrypi kernel: [11180.089888]  r4:82211800
Jan  4 20:47:58 raspberrypi kernel: [11180.089909] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:48:03 raspberrypi kernel: [11180.089922]  r5:82211000 r4:82211000
Jan  4 20:48:14 raspberrypi kernel: [11180.089942] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:48:20 raspberrypi kernel: [11180.089954]  r5:82211000 r4:8221127c
Jan  4 20:48:29 raspberrypi kernel: [11180.089976] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:48:43 raspberrypi kernel: [11180.089993]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:48:52 raspberrypi kernel: [11180.090013] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:49:01 raspberrypi kernel: [11180.090031]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:49:08 raspberrypi kernel: [11180.090042]  r4:86abf900
Jan  4 20:49:18 raspberrypi kernel: [11180.090061] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:49:30 raspberrypi kernel: [11180.090080]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:49:39 raspberrypi kernel: [11180.090091]  r4:9e4771c0
Jan  4 20:49:53 raspberrypi kernel: [11180.090110] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:50:00 raspberrypi kernel: [11180.090122] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:50:11 raspberrypi kernel: [11180.090136] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:50:24 raspberrypi kernel: [11180.090153] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:50:38 raspberrypi kernel: [11180.090167] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:50:48 raspberrypi kernel: [11180.090185]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:51:03 raspberrypi kernel: [11180.090196]  r4:b6004780
Jan  4 20:51:20 raspberrypi kernel: [11302.970915] INFO: task kworker/2:3:12050 blocked for more than 860 seconds.
Jan  4 20:51:38 raspberrypi kernel: [11302.970938]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:52:01 raspberrypi kernel: [11302.970948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:52:15 raspberrypi kernel: [11302.970960] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:17:06 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0

Any way we can find out what is causing this ?

titiviking commented 3 years ago

I was able to solve the issues, but it required a small investment...

I did the following:

  1. Use the power adapter provided, it reduces slightly the occurences, but don't fully remove them... You can keep it active for slightly longer, but after a while errors occur again and you need to reboot it by power-cycling
  2. Get rid of SD-card. SD cards are slow, so I went for a USB 3.0... works a bit better, but the USB is still slow (especially in my case where Home Assistant writes to the DB quite a bit)
  3. Get rid of USB and go for SSD. This is magical!!! No more errors, no more latency, no more unavailability. Going SSD was THE solution for me. Raspberry is now fast and never got stuck again. In order to attach the SSD I got an adapter board from amazon .
  4. Next step for me if an upgrade to a Raspberry Pi 4 B that has USB3.1 ports to go even faster and really have seamless experience and reduced lag when it comes to reading large amounts of values from the HA DB.
All3xJ commented 2 years ago

I still have this bug with 5.15.30-v8+ 64 bit Raspberry Pi 3B+.... the system freezed and resumed after 3 hours of freeze. Here are journalctl logs: journalctl.txt

pelwell commented 2 years ago 
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Undervoltage detected!
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Voltage normalised

Your system is struggling because there is insufficient power getting to the processor.

  1. Check the power supply - it should be capable of delivering at least 3A for a Pi 4. Note that a charger is not necessarily the same as a supply because chargers don't have to worry about voltage and current variations.
  2. Check the cable from the supply to the Pi - it's shocking how much power can be lost by in-line switches etc.
  3. If you have power-hungry USB peripherals attached, put them on a powered hub.
All3xJ commented 2 years ago 
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Undervoltage detected!
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Voltage normalised

Your system is struggling because there is insufficient power getting to the processor.

1. Check the power supply - it should be capable of delivering at least 3A for a Pi 4. Note that a charger is not necessarily the same as a supply because chargers don't have to worry about voltage and current variations.

2. Check the cable from the supply to the Pi - it's shocking how much power can be lost by in-line switches etc.

3. If you have power-hungry USB peripherals attached, put them on a powered hub.

I have these logs all the time, always.... and it never happened that it completely freezed for 3 hours, so I don't think that this is the real reason. I followed this workaround: https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/

sresam89 commented 4 months ago

I can confirm the same occurs on newer firmware too. Started happening after I migrated from 32-bit to 64-bit OS

RPI4 2GBRAM & its original power supply Using Brand new 64GB samsung evo sd card. (Switched to a new card since 64 bit OS crash started) Default partition size done by RPiOS flash tool. No Wifi, hardwired to network. No USB accessories, running headless (not while debugging the hanging state)

pi@raspberrypi:~ $ docker-compose ps
NAME            IMAGE                                                     COMMAND                  SERVICE         CREATED      STATUS       PORTS
homeassistant   ghcr.io/home-assistant/home-assistant:stable              "/init"                  homeassistant   5 days ago   Up 6 hours
matterServer    ghcr.io/home-assistant-libs/python-matter-server:stable   "matter-server --sto…"   matterServer    6 days ago   Up 6 hours
mosquitto       eclipse-mosquitto:latest                                  "/docker-entrypoint.…"   mosquitto       6 days ago   Up 6 hours
neolink         quantumentangledandy/neolink                              "/entrypoint.sh /usr…"   neolink         4 days ago   Up 6 hours
pi-mariadb-1    yobasystems/alpine-mariadb:latest                         "/scripts/run.sh"        mariadb         6 days ago   Up 6 hours   3306/tcp
pi-nginx-1      jc21/nginx-proxy-manager:latest                           "/init"                  nginx           6 days ago   Up 6 hours   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp

Occurs every two to three hours, hangs up and waits for forced reboot via removing power. Console, network-ping, everything becomes unresponsive.

Tried the solutions recommended in this post

I've so far found some advice from https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/ summary: sudo sysctl -w vm.dirty_ratio=10 sudo sysctl -w vm.dirty_background_ratio=5 sudo sysctl -p seems to do the trick so far, still testing (3 movies down) I'll comment if it occurs again. To make it permanent(after reboot) read the link

no luck there

pi@raspberrypi:~ $ free
               total        used        free      shared  buff/cache   available
Mem:         1892500      815472      196372        2212      951516     1077028
Swap:         102396           0      102396
pi@raspberrypi:~ $

uname -a
Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux
pi@raspberrypi:~ $ vcgencmd version
Oct 17 2023 15:39:16
Copyright (c) 2012 Broadcom
version 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean) (release) (start)
pi@raspberrypi:~ $ cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2
journalctl -b -1
Mar 02 23:23:03 raspberrypi kernel: INFO: task kworker/0:3:2023 blocked for more than 120 seconds.
Mar 02 23:25:37 raspberrypi kernel:       Tainted: G         C         6.1.0-rpi7-rpi-v8 #1 Debian 1:6.1.63-1+rpt1
Mar 02 23:26:42 raspberrypi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 02 23:27:46 raspberrypi kernel: task:kworker/0:3     state:D stack:0     pid:2023  ppid:2      flags:0x00000008
Mar 02 23:29:30 raspberrypi kernel: Workqueue: events_freezable mmc_rescan
Mar 02 23:33:15 raspberrypi kernel: Call trace:
Mar 02 23:35:23 raspberrypi kernel:  __switch_to+0xe8/0x160
Mar 02 23:48:33 raspberrypi kernel:  __schedule+0x338/0x924
Mar 03 00:44:29 raspberrypi kernel:  schedule+0x64/0x10c
Mar 03 01:31:32 raspberrypi kernel:  __mmc_claim_host+0xe0/0x204
Mar 03 02:31:38 raspberrypi kernel:  mmc_get_card+0x3c/0x4c
journalctl -b -2
Mar 02 13:29:46 raspberrypi kernel: INFO: task kworker/0:0:2138 blocked for more than 120 seconds.
Mar 02 13:32:48 raspberrypi kernel:       Tainted: G         C         6.1.0-rpi7-rpi-v8 #1 Debian 1:6.1.63-1+rpt1
Mar 02 13:36:04 raspberrypi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 02 13:39:09 raspberrypi kernel: task:kworker/0:0     state:D stack:0     pid:2138  ppid:2      flags:0x00000008
Mar 02 13:41:02 raspberrypi kernel: Workqueue: events_freezable mmc_rescan
Mar 02 13:43:13 raspberrypi kernel: Call trace:
Mar 02 13:44:47 raspberrypi kernel:  __switch_to+0xe8/0x160
Mar 02 13:45:52 raspberrypi kernel:  __schedule+0x338/0x924
Mar 02 13:46:58 raspberrypi kernel:  schedule+0x64/0x10c
Mar 02 13:48:54 raspberrypi kernel:  __mmc_claim_host+0xe0/0x204
Mar 02 13:50:56 raspberrypi kernel:  mmc_get_card+0x3c/0x4c
Mar 02 13:52:55 raspberrypi kernel:  mmc_sd_detect+0x28/0xa0
Mar 02 13:55:59 raspberrypi kernel:  mmc_rescan+0x94/0x2ec
Mar 02 13:59:29 raspberrypi kernel:  process_one_work+0x200/0x474
Mar 02 14:02:55 raspberrypi kernel:  worker_thread+0x74/0x43c
Mar 02 14:06:27 raspberrypi kernel:  kthread+0x110/0x114
Mar 02 14:09:28 raspberrypi kernel:  ret_from_fork+0x10/0x20