search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.05k stars 4.97k forks source link

Kernel/Firmware update bricks multiple RPiZero Ws. #4007

Open yik77 opened 3 years ago

yik77 commented 3 years ago

I believe this is right place (or one of) to report this issue. It is potentially very high impact issue.

Simply put, upgrading to latest stable and default version of kernel/firmware reproducibly causes frozen system (LED sometimes keeps blinking-7 blinks) and results in a bricked, unbootable device. screenshot

I understand your healthy skepticism with this statement, but I did verified everything I can, before voicing this strong statement. Let me elaborate:

I have tested it with MULTIPLE brand new SD cards, [Sandisk, bought in a brick and mortar store] with MULTIPLE power supply options [Canakit official power supply unit, a table lamp with USB, a power bank, and finally with laboratory electronic power supply unit. I have tested it with "old" and with another "new" pi zero w board, so with 2 different pi zero w boards, with 2 completely separate devices. Same faulty behavior. I have been monitoring them by having my HDMI monitor connected in, and watching boot messages and system log. I have also replaced power cables few times. There are NO peripherals on either of my pi zero w boards. No keyboads, no USB, nothing attached. Nothing put power.

I have tested it with my "old" pi zero w, running very light load, mostly pihole, dnscrypt-proxy and local intranet clipboard. This device was rock stable and I have seen maybe 4-5 kernel level crashes in the last 2 years, so 5x in last 500+ days. In last few days, however, with new kernel released, during/after upgrading, it start failing on constant basis. Often during the upgrade itself or during the first consequent boot. Very bad situation.

I do have 2 more pictures of kernel panic, it may be related to wifi firmware/drivers/kernel/kernel modules. I believe this is rather important finding, which may possibly impact many users and hit raspberry ecosystem...

2 more screenshots are crash during apt-get update and crash of freshly burned Pi OS lite image during first boot

To reproduce install latest package with kernel/firmware

System plain vanilla Pi OS lite image

timskillman commented 3 years ago

I can confirm something similar with several Raspberry Pi Zeros (both Wifi and non-wifi) ... both types will shutdown after an hour or so (running in X11, firmware 5.4.79+). Reverting to earlier OS (X11, firmware 4.14.98+) the Pi Zero's are rock solid. I have tested on 2x PiZero W's and 3x non-wifi Pi Zero's.

I am developing with SDL2 and enable the sound drivers so I have a sneaking suspicion that it could be the new sound drivers causing the issue ... only because ALSA lib used to have underrun's on my Zero's after a while. Perhaps something similar happens with the new sound drivers but they cause a shutdown instead?? Anyway, just a hunch - but I'll do some more testing by disabling sound drivers etc...

yik77 commented 3 years ago

I am not sure if this does or doesn't have something to do with the instability, but I expect things like that are NOT supposed to be in the log...

Dec 21 02:17:13 newpizero kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
Dec 21 02:17:13 newpizero kernel: usbcore: registered new interface driver brcmfmac
Dec 21 02:17:13 newpizero kernel: brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43430-sdio.raspberrypi,model-zero-w.txt failed with error -2
Dec 21 02:17:13 newpizero kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
Dec 21 02:17:13 newpizero kernel: brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available
Dec 21 02:17:13 newpizero kernel: brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: Oct 22 2019 01:59:28 version 7.45.98.94 (r723000 CY) FWID 01-3b33decd
HattonLe commented 3 years ago

Had Pi Zero W h/w running old "2019-09-26-raspbian-buster-lite" OS working 100% ok. Did an OS update to latest "2021-01-11-raspios-buster-armhf-lite" but when git cloning the OS rep it always crashed out with kernel panic. Another time Pi just paniced on its own. Looked on web, and tried suggestion "force_turbo=1" in boot/config.txt. No panics as yet, so maybe worth a try.