procount / pinn

An enhanced Operating System installer for the Raspberry Pi
1.14k stars 123 forks source link

Kernel panic after update from 3.8.8 to 3.9.1 #815

Open VortexNexus opened 6 months ago

VortexNexus commented 6 months ago

Hello,

I've been using PINN since years now and its a really great solution, always improved in a very good way. I never had any problem during installation nor update until today.

On a fully functional Raspberry PI 3B v1.2 (so quite an old one for sure) running PINN 3.8.8, I have been notified a new version was available, and as usual I applied the upgrade. After that, PINN didn't start any more, displaying a Kernel panic message. Additionally, the "os" sub-directory was empty, but I usually put there the image(s) of the installed system(s) as I am currently doing tests often resulting in a dysfunctional bookworm system, and so I need to be able to replace it from scratch (thanks to PINN functionalities).

Some details to reproduce:

To solve my problem, I re-formatted the RECOVERY partition, restored PINN 3.8.8 files and its "os" sub-directory content, updated the recovery.cmdline file (essentially removing the "runinstaller" argument, and adding the "no_update"). Then, PINN and the two RasPIOS systems were able to start again normally.

I know PINN is currently on its way to gain full compatibility with the PI 5. So perhaps something is simply wrong in my setup (using an old PI 3B). In that case, just let me know about it.

Thanks a lot for your work.

procount commented 6 months ago

Sorry the upgrade didn't quite work out for you. The new version is quite a bit bigger to accommodate the Pi5 kernel and I struggled to get it to fit (hence the change in wallpaper for some cases). I tried hard to account for the change in firmware names etc, but a few corner cases slipped through. So for ease in 3.9.1 I just wipe the recovery partition and install the new version. The only thing that is preserved are the settings from recovery.cmdline to the new cmdline.txt. This is why your /os folder contents disappeared.

You already have an expanded recovery partition, so I'm surprised it failed (unless your /os folder was totally full!). You could repeat your repair process using PINN 3.9.1 instead of PINN 3.8.8 and see if that works (if not, just go back to 3.8.8). Make sure you have enough free space for it (maybe delete the /os folder temporarily and restore afterwards) 3.9.1 is essentially 3.8.8 PLUS the Pi5 kernel and overlay files so there shouldn't be any reason for the kernel panic on a PI3. Maybe it was just an installation failure of some sort 🤷‍♂️. Let me know if it's still a problem though.

Glad to hear you are making the most of PINN's functionalities, though! 😁

VortexNexus commented 6 months ago

My recovery partition has about 700Mb free space even with the /os sub-directory containing RasPIOS images. So I don't think the partition space was causing the problem.

As you proposed, I retried my repair process using PINN 3.9.1 version and modifying the /cmdline.txt file (thanks to you pointing the name change). As you expected, everything went well. But now I don't know what went wrong the first time, and I don't dare let the update happen on my other raspberry pi.

You surely noticed the unnatural data partition size. After OSes installation (through PINN), I used GParted to move and resize all partitions in the way I need for my tests. But I didn't "inform" PINN about those changes in any way. Do you think this could be the origin of the update problem? If so, what should I do so that PINN can deal with the changes during the update process?

Additionally, do you plan from now on to force this /os sub-directory "clean up" on every PINN's update? This would be very annoying as I don't have physical access to every raspberry pi I use, and keeping the original OSes images on the same SD card occasionally saved my life ;-) ...

procount commented 6 months ago

I can't imagine what happened, then. The upgrade process normally works fine. From 3.8.8 I added a 'preupdate' script feature so that I could adapt the upgrade process. This was necessary this time to account for the filename changes and the lack of disk space. In addition, I save the boot partition contents to a temporary folder before upgrading so I can rollback if it fails due to lack of disk space. Hmm, a thought has just occurred to me that this might have caused your failure as you have a large /os folder. Could you perhaps test that hypothesis? If you restore 3.8.8 with and without your /os folder populated and see if that makes a difference to the upgrade to 3.9.1? Provided you have local access to your RPi to recover if it fails again, you should be ok as the /settings partition and all your OSes are left untouched. To be extra safe, just take a backup copy of your /os, config.txt and cmdline.txt/recovery.cmdline files before upgrading. BTW - it shouldn't be necessary to reformat the RECOVERY partition - just delete all the files on it before unzipping pinn-lite.zip.

Moving and resizing the partitions should not have any effect on PINN. PINN only cares about the order of the partitions and depending on the OS, the partition label, partuuid or uuid. It does not care about the size or location of the partition. PINN 3.9.1 now has the ability to specify the partition sizes when you install the OSes (similar to Matt's webpage) so that should make life easier for you!

I shall have to rethink the "clean up" of the boot partition on the next upgrade. I had overlooked that as I am stopping providing a full NOOBS image with some OSes stored in the /os folder as it is too time consuming to produce. For users who have already enlarged their recovery partition, this should not be necessary anyway.

VortexNexus commented 5 months ago

I made the tests you asked for, with and without my /os folder populated. Just to remember, here are the steps I applied: (hardware: empty 64Gb SD card, Raspberry Pi 3B rev1.2, Raspberry Pi 3B+ rev1.3)

In conclusion, your feeling seems to be right. When the /os folder is empty, everything goes well. But when there is something in the /os folder, the update process ends on a kernel panic message: KernelPanic

Beside this, I have bad news: I tried to update to PINN 3.9.1 on a Raspberry Pi 4 (Model B Rev 1.4). I tried through the normal PINN 3.8.8 update process, and also manually replacing the files in the recovery partition. This Pi has its OSes images on a USB key, and it does not have any data in the /os folder. However there is a reserve=+800 cmdline argument, and so the recovery partition has enough free space. But I've not been able to start PINN 3.9.1, only obtaining the following lines: AppCrash So I manually made a roll-back to PINN 3.8.8.

I don't have any Raspberry Pi 5 at the moment, so I couldn't give it a try.

Hope all this will give you a little help.

procount commented 5 months ago

Thanks for doing these tests. I think that confirms my suspicion and the the kernel panic is understandable. The issue with your 4B looks like something else related to the display. Do you have a monitor plugged into the 2nd HDMI port instead of the 1st one next to the USB-C power input?

EDIT: Have you modified config.txt or pinn_init? What happens if you rename pinn_init? e.g. mv pinn_init pinn_init.bak

VortexNexus commented 5 months ago

Effectively, changing the HDMI port allowed a successful update process to 3.9.1, like expected as /os folder is empty.

No I didn't do any change in PINN's configuration, except in the recovery.cmdline file (adding allowed parameters like vncshare and others). Those changes where already effective for PINN 3.8.8.

I found that those kind of messages are displayed by an old feature called RPI safe mode, only still supported by NOOBS (and possibly by inheritors like PINN), that can be activated by shunting GPIO pins 5 and 6, to be able to edit config.txt and cmdline.txt with vi in a busybox shell. My pi is a media server boxed in an Argon ONE M.2 Case, but as far as I can see, no GPIO pin is in use.

So, that's right: my HDMI connection was not "as expected", but falling in emergency mode seems to me a little extreme :-) and not really expectable as PINN 3.8.8 is not affected by the misconnection. Nevertheless, thanks to you for your really appreciate help and work.

procount commented 5 months ago

Wow, that was totally unexpected. I don't think it is a failure of the upgrade process, but a failure of 3.9.1 to startup when using HDMI2 instead of HDMI1.

It's not from RPi Safe mode, but when the Pi4 came along with 2 HDMI ports, I tried to detect which one is in use and switch PINN to using that one. First time I have seen such a failure. I will see if I can replicate it.

procount commented 5 months ago

This is indeed an issue caused by the renaming of the recovery files. This was necessary as the Pi5 does not support this alternate load mechanism, but the loss of HDMI2 use was totally unexpected. The names of the firmware files seem to change their behaviour. I have reached out to see if I can get some help fixing this. Meanwhile, don't use HDMI2 !

procount commented 5 months ago

Fortunately, we have a quick solution! 😄 Please add the following to config.txt, I suggest just above the [pi4] section:

[HDMI1]
hdmi_force_hotplug=1

This is done automatically in recovery.elf, but when it's renamed to start.elf, it isn't.

Pivan78 commented 5 months ago

Same problem as above. But on Raspberry Pi 4 2GB working normally. If I installed PINN older version to Raspberry Pi 400, working well. But after update got error:

ioctl FBIOPUT_CON2FBMAP: Invalid argument ioctl FBIOPUT_CON2FBMAP: Invalid argument
Recovery application crashed Starting shell sh: can't acces tty; job control turned off /sys/class/block #

After exit - kernel panic... Hardware name BCM2711

Doesn't matter if is fresh install or update from older version. Or if boot from SDcard or USB. Old version working without issues.

procount commented 5 months ago

Did you modify config.txt as above?

Pivan78 commented 5 months ago

Did you modify config.txt as above?

Yup. But not help in my case. I stay with previous version for now.

procount commented 5 months ago

Are you using the secondary HDMI port? Does it fail if you use the other HDMI port?

procount commented 5 months ago

I've tested the change on my Pi400 and it works fine. Please check you modified it correctly. I am pushing this change up in v3.9.2 shortly.

procount commented 5 months ago

@VortexNexus - The v3.9.2 upgrade should now work with your expanded recovery partition and populated /os folder.

dbjones-clanjones commented 5 months ago

Although I had been aware of PINN for a little while, but never had gotten around to using it, probably as I use all my Pi 4b's headless for server apps and my Pi 400 has gathered dust for the last year as I had to undertake multiple house moves. However, I want to be able to use to try multiple different operating systems such as the latest Ubuntu and Fedora, without messing with my Raspberry Pi OS install all from the same Samsung T1 500gb SSD drive. So I began trying yesterday, but have run into problems. I started with PINN 3.9.1 and I am receiving the same Kernal panic error when I create 3.9.1 from Raspberry Pi Imager on a Raspberry Pi 400, using a Samsung T1 500gb SSD drive (also tried using Sandisk 256gb USB 3.1 key). It booted successfully from the same drive(s) using PINN 3.8.8. Some (or possibly all of) these attempts had 2 HDMI monitors connected. I'll try again and if problem persists will create a separate issue. But wanted to log that the Kernal Panic had happen on my Raspberry Pi 400.

Tried again with 3.9.2 and the issue is no longer occurring.

procount commented 5 months ago

Glad to hear 3.9.2 fixed your issues. But I'd like to understand your 3.9.1 issue a little more as 3.9.2 fixed 2 bugs.

You said you started with 3.9.1 and got a "kernel panic" after flashing it directly from Rpi Imager. If so, that would be a different issue because the kernel panic mentioned here was due to an upgrade failure from 3.8.8. "Kernel panic" is actually mentioned on the screen.

But you also mentioned you tried 3.8.8. If you upgraded from there to 3.9.1, the kernel panic would occur due to additional files on the recovery partition like in the /os folder..

The 2nd bug fixed in 3.9.2 concerns using the 2nd hdmi port, but I thought it would only happen if you only had hdmi1 connectrd. Switching to hdmi0 would solve that issue. Not sure it would occur if you had both connected at the same time. This was not a kernel panic.

VortexNexus commented 5 months ago

@procount: Sorry for the late answer, and for the the bad news, but it seems this issue cannot be closed at the moment :-) ...

I made the tests you asked me for again, trying to update from PINN 3.8.8 to 3.9.2. But it seems that this requires PINN to update first to 3.9.1, falling into the same problem of the kernel panic. In addition, and this is my bad as I didn't noticed it during my first tests, it appears that when os folder has any content, the update process from 3.8.8 to 3.9.1 clears its content but also clears the cmdline.txt file content ... Now we have the real origin of the kernel panic fallback. And, bad news again, the update process from 3.9.1 to 3.9.2 keeps the os folder (thanks a lot for the modifications you made), but clears the cmdline.txt file content too, and so fallbacks again into kernel panic. This appears logic to me as I imagine you didn't change anything in 3.9.2's cmdline.txt handling, so the problem has been inherited. On the other hand, when os folder is empty, everything goes well in the chained update processes, so there are good news too: no unexpected modifications' side effect :-)

procount commented 5 months ago

Hopefully the update to 3.9.2 fixes things now.

VortexNexus commented 5 months ago

Mmh, not sure to understand what you mean. As explained in my previous post, 3.9.1 and 3.9.2 updates both left the cmdline.txt file without any content ...

I thought it was possible to update from PINN 3.8.8 directly to 3.9.2 using the "Ignore" button on PINN's 3.8.8 update confirmation dialog. So I tried, but it seems it has the same behavior as the "No" button, doing nothing.

Now, imho, for anyone having PINN 3.8.8 installed with a non empty os folder, auto-update is not possible anymore, leaving only 2 choices:

Do you see another option?

procount commented 5 months ago

Assuming you have 3.8.8 installed, please try the following:

  1. Download https://sourceforge.net/projects/pinn/files/pinn64/pinn-lite.zip/download
  2. With PINN booted, copy this file to /tmp. (This is a temporary folder, so if you turn off your Pi or reboot it, it will be lost) You can do this several ways: a) transfer it via USB stick. You'll need to do it from an SSH shell or the recovery cmd shell (login with root/raspberry). Or b) my favourite using scp: scp pinn-lite.img root@<PINN-ip-address>:/tmp (replace \<PINN-ip-address> with the IP address of your Pi)
  3. Go to PINN's maintenance menu
  4. tick the checkbox next to PINN and select reinstall. This will reinstall PINN from the 3.9.2 pinn-lite.zip file in the /tmp folder, thus hopefully bypassing your problems with 3.9.1. 🤞
VortexNexus commented 5 months ago

I've given it a try, but unfortunately, that didn't work.

Remarks:

At the end of the transfer process, I checked through SSH that the archive file was really present in the remote /tmp folder. So, having all set up, I went to the maintenance menu, toggled the PINN's line check box, and clicked the "Reinstall" button. The PINN's update dialog showed up. A click on the Details button showed the 3.9.1 change log ... Anyway, I thought the data have been downloaded from internet, so I started the update ... and obtained the now well-known kernel panic message :smile:.

So it seems the update process don't care about an already downloaded ".zip" file in the /tmp folder. Or perhaps, do I have to rename the ".zip" file into ".img"? Seems not so obvious ...

iNgeon commented 4 months ago

Have a Raspberry PI4 4GB Model B. Everything was working fine. PINN version p3.8.7 was installed PINN asked for update on boot. Did first updgrade and PINN upgraded to p3.8.8 Rebooted. PINN asked for update again. PINN updated to p3.9.2 Raspberry rebooted with console on screen, crash-failed with last line item. end Kernel Panic Formatted SD card and started from scratch iow wiped all other partitions and only left a 16GB FAT32 for PINN to copy. Downloaded latest version of v.3.8.8 pinn,.zip 2024-02-19, 3.4GB SD format. Again PINN asked for update to p3.9.2 Raspberry rebooted with console on screen, crash-failed with last line item. end Kernel Panic Note: Also tried both HDMIO and HDMI1

Attached 20240628_162104 screenshot

procount commented 4 months ago

Unfortunately the log doesn't help me much. It says it can't mount the rootfs, which is actually an initramfs so I'm confused. As you have wiped your card, why don't you run RPi Imager and install PINN 3.9.2 directly from there onto your SD card? You'll find PINN under the Misc Utility Images.

iNgeon commented 4 months ago

Can't seem to find a log ? Does PINN dumb one somewhere on the disk ? Will try loading 3.9.2 direct, was hoping to share the full log, tried capturing on video via phone but just a blur.

edit: Okay loaded onto SD card, 3.9.2 worked by installing the Lite version, still would like to pull full log on upgrade fail, i have some other PI4 i'd like to upgrade PINN without having to redo everything...

image

procount commented 4 months ago

There is the kernel dmesg log which you screenshot a photo of, and PINN has its own log in /tmp/debug. Unfortunately, neither of these are saved if you lose power, so in the case of a kernel panic you can't get at them. If the upgrade fails to the command prompt, then you can usually access the logs before you remove the power.

The kernel panics during the development of the upgrade process were due to additional files in the /os folder, but that was resolved and you've tried an upgrade from a default 3.8.8 version (which worked for me) so this is something new. Is there anything out of the ordinary with your setup? - any unusual hardware attached, for example, or additional files stored on PINN's recovery partition other than those supplied?

Something to remember in future, is that if an upgrade fails, it will only really affect the recovery partition, so there's no need to completely start from scratch and wipe your OS partitions. In that case, the best way to proceed is to wipe only the RECOVERY partition and then unzip pinn-lite.zip (v3.8.8) onto it. When you boot PINN, there will be a prompt to delete all the partitions, but you should skip that to keep your existing OSes. (You can avoid that risk altogether by deleting the 'runinstaller' option from cmdline.txt to be doubly sure). 3.9.2 is really only the merger of the PI5 beta with the other Rpi models. The only thing you're getting extra is the adjustable partition sizes, so if you're happy without that, you can stick with 3.8.8 until this is resolved. The other major change is the recommended recovery partition size has doubled to 128MB to account for the new kernel. 3.9.2 will just about fit in 64MB but it is a tight squeeze. The upgrade process tries to make sure it will fit (e.g. by using the smaller blue wallpaper) and will rollback to the old version if it won't, but maybe something on your system is breaking it.

In order to resolve this, seeing as there are no logs, I think it would be necessary to have an image of your recovery partition so I can debug this myself (provided you haven't filled your /os folder with MBs of files!). So before you upgrade another one, please take an image of just the RECOVERY partition (e.g. use dd) and a copy of the output of fdisk -l so I can construct an identical image BEFORE upgrading. (Ask me if you don't know how to do this). NOTE: if the upgrade fails, this image would be useful to restore your card back to where it was beforehand anyway as an alternative to reinstalling 3.8.8.

At some point, if you are upgrading several PINN installations, you should really expand the RECOVERY partition otherwise future PINN upgrades may not fit.

  1. Selecting the runinstaller option is one way to do this, but the downside is it will wipe all your OSes. One way to overcome this is to use PINN to backup each OS to a USB stick first. Install PINN 3.9.2 to a new SD card of similar or larger size and then reinstall all your backed -up OSes from USB. This is a bit long-winded and will take a while, but it keeps your original card intact until you have verified your backups have restored correctly. If not, you still have your original SD card as a backup! If it works, you can reuse your original SD card for the next SD card to upgrade.
  2. Another way is to use Gparted to move and resize your partitions. Your PINN SD card must not be mounted to do this so you will need to do it on another computer, or by running another OS. To minimise the disk moving required (assuming a typical installation), I would shrink P7 by 64MB, move it right, Move P6 right, then move P5 right. This will leave you with a gap at the start of the extended partition P2. Now you can shrink P2 by moving the start of it right 64MB, and finally you can expand P1 to fill the gap. (It's a but like one of those sliding puzzle games!) Also you can do this on a clone of your SD card, so if something goes wrong, you still have the original SD card.
VortexNexus commented 4 months ago

@procount : as I pointed it in a previous post, the update process to 3.9.2 clears the content of the cmdline.txt file. I'm not sure you noticed it. For me, this is the reason why the rootfs cannot be mounted (the system has no info about which mounting point must be used), and also the reason why the system falls in the emergency "kernel panic" mode.

procount commented 4 months ago

@VortexNexus - yes I noticed your comment and changed the procedure. The upgrade process (should) now copy recovery.cmdline (or cmdline.txt) to /tmp first, then restores it after upgrading. Rather than reinstalling a totally new version, I maintain the old version and modify the few changes that are needed for 3.9.2. I didn't experience any failures after implementing this, but if yours is still failing, then clearly some use case is still missing and it needs relooking at. @iNgeon - If you get another failure again, please check if cmdline.txt is present after the upgrade.

mik-ohdsi commented 2 months ago

I got stuck in the same trap. After a long while entered the PINN GUI again, upgraded to 3.8.8 and after restart was notified of yet another upgrade. After doing that I now have the aforementioned ioctl error. Is there anything I could do from the console itself now or is it only shutting down and putting the pinn-lite.zip into the recovery folder after prying the SD card from my Pi?

procount commented 2 months ago

Sorry to hear this happened to you. I'm afraid once you get a kernel panic, the cmdline is inoperative, so you will need to pry the SD card from your Pi. Once you do, please check if cmdline.txt exists, and it is not empty. If it isn't there or is empty, try restoring it to: repo_list=http://raw.githubusercontent.com/procount/pinn-os/master/os/repo_list.json quiet ramdisk_size=65536 root=/dev/ram0 init=/init vt.cur_default=1 elevator=deadline loglevel=2 sdhci.debug_quirks2=4 I would also be interested to know if cmdline.bak, config.txt and config.bak exist and what their contents are. Please let me know if restoring cmdline.txt fixes this for you. If not, we shall try something else.

mik-ohdsi commented 2 months ago

cmdline.txt exists and contains the following: quiet ramdisk_size=65536 root=/dev/ram0 init=/init vt.cur_default=1 elevator=deadline repo_list=http://raw.githubusercontent.com/procount/pinn-os/master/os/repo_list.json loglevel=2 sdhci.debug_quirks2=4 So, basically the same content, but in a different sequence. cmdline.bak is identical. config.txt looks like this: gpu_mem=16 start_file=start.elf fixup_file=fixup.dat

disable_overscan=1 initramfs pinn.rfs

[pi4] start_file=start4.elf fixup_file=fixup4.dat max_framebuffers=2

[board-type=0x17] kernel=kernel8.img overlay_prefix=overlays6/ dtoverlay=vc4-kms-v3d max_framebuffers=2

config.bak looks considerably different: gpu_mem=16 start_file=recovery.elf fixup_file=fixup_rc.dat

[pi4] start_file=recover4.elf fixup_file=fixup4rc.dat max_framebuffers=2

The "os" folder is empty and there is no file "recover4.elf" in the RECOVERY partitition.

The SD Card has been prepared in an Apple computer and hence has the hidden Spotlight folder and a couple of those files starting with ._ that contain information for MacOS about files in the folder. I can see that previously there has been a recover4.elf in this partition.

I tried it with a cmdline.txt with your sequence of settings, but... not unexpectedly, that had no effect. So, course of action: clear the entire RECOVERY partition and place the pinn-lite.zip from sourceforge or the pinn-lite.img.zip from GitHub there instead?

Thanks for your advice!

procount commented 2 months ago

The problem with upgrading from 3.8.8 to 3.9.x is that PINN now includes support for Pi5 with its extra kernel image and overlays and it only just fits - we're talking only kilobytes or even bytes spare! Those extra Apple related files may have made the difference in preventing some files from being copied. You cannot install pinn-lite.img.zip without removing all your OSes because this image is twice as big and will not fit in the existing partition size. Likewise, just unzipping pinn-lite.zip even to an empty RECOVERY partition, will probably still not fit because it has some unwanted files.

My suggestion would be to:

  1. Delete all files on your RECOVERY partition. (maybe keep your cmdline.txt and config.txt files if they are correct, and especially if they have any of your own customisations).
  2. Copy the contents of pinn-lite.zip to the recovery partition, with the exception of kernel8.img and wallpaper.jpg (and config.txt & cmdline.txt if you kept them from above)
  3. Rename wallpaper2.jpg to wallpaper.jpg.

NOTE: the RECOVER.elf files have now been renamed to the more usual START.elf names.

This should get you up and running again, but the omission of kernel8.img means it won't run on a Pi5 until it is restored.

In the longer term, I advise you to increase the size of your RECOVERY partition to allow for future upgrades. You can do this in several ways:

mik-ohdsi commented 2 months ago

Ah, that is confusing. The RECOVERY partition is currently at 160MB with 96MB free space at this time (with all files left on it). I may have reserved enough space from the get go... This is a Pi4 4GB with a 32GB SD Card. And the config.txt mentions kernel8 but I should rather omit it? Should a Pi4 use only kernel7? The os folder currently is empty but should it contain something which may have been deleted during the upgrade process?

procount commented 2 months ago

Oh ok. I don't understand how it is so big because PINN would shrink its partition on first boot. In that case, just unzip the whole of pinn-lite.zip to the RECOVERY partition. Remember to remove runinstaller from cmdline.txt before booting for the first time. If you forget, it will recognise that you already have some OSes installed and ask you if you really want to reformat the SD card, in which case say NO.