official rpios bookworm issue

[ProblemChild4901]

The official rpios bookworm release has changed the mount point /boot to /boot/firmware. The current rpi-clone can back up, but the backup media is not bootable because the cmdline.txt file is not updated with the new PARTUUID.
/boot/cmdline.txt -> /boot/firmware/cmdline.txt Consider adding lines like this after these variable assignments "cmdline_txt=${clone}/boot/cmdline.txt"

if [ -h $cmdline_txt ] ; then cmdline_txt=${clone}/boot/firmware/cmdline.txt fi

This tests if the /boot/cmdline.txt is a link and changes it to the actual file.

[framps]

Does my proposed line work in any case also on a fresh install?

As far as I understand - yes. But I will switch to --follow-symlinks now. See my previous reply.

[bputtick]

When cloning a few SD cards in a row I've noticed that /boot/firmware becomes unmounted and then rpi-clone doesn't update the UUID at all, so I had to make sure that it was mounted each time.. I assume rpi-clone is unmounting it ?

[fmarzocca]

@bputtick , AFAIK rpi-clone is not unmounting any current partition! Just the ones it clones to.

[bputtick]

@bputtick , AFAIK rpi-clone is not unmounting any current partition! Just the ones it clones to.

Yeh I've not really investigated it, but it happened more than once and I was just cloning, replacing SD card and repeating and nothing else.

Just thought I'd mention it, in case someone else experienced it.

[framps]

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

[bputtick]

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

I felt this was the safest fix, at least until they remove the symlink they put in for backwards compatibly

[fmarzocca]

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

But line 1730 is still there, with wrong syntax...

[fmarzocca]

...and it is still modifying original file!!!

Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

[fmarzocca]

I have re-instated my correct line 1730, and it is working back again:

Editing /mnt/clone/boot/firmware/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

[framps]

But line 1730 is still there, with wrong syntax...

:+1: Good catch. I now added --follow-symlinks in every sed call.

[fmarzocca]

but if you "follow symlink" on the cloned partition, it will point to /boot/firmware and NOT to /mnt/clone/boot/firmware

because of this: lrwxrwxrwx 1 root root 26 Dec 29 13:53 cmdline.txt -> /boot/firmware/cmdline.txt

[fmarzocca]

In this line cmdline_txt contains /mnt/clone/boot/cmdline.txt

sed -i --follow-symlinks "s/${src_disk_ID}/${dst_disk_ID}/" "$cmdline_txt"

and when you follow that symlink, it will then point to /boot/firmware/cmdline.txt

Why you didn't just keep your previous version and change that 1730 line?

[framps]

Ok. So you're saying the --follow-symlink approach does not work for you and thus realpath is the only way to fix your issue?

I unfortunately have no upgraded bookworm in order to test your scenario :cry:

Why you didn't just keep your previous version and change that 1730 line?

Looks like I should revert the --follow-symlinks approach ...

[fmarzocca]

Looks like I should revert the --follow-symlinks approach ...

Why, if your previous version was working?

[fmarzocca]

Can you send me the output of ls -la /booton a fresh installed bookworm?

[fmarzocca]

Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

As you can see, it is NOT following to /mnt/clone/boot/firmware

[framps]

Can you send me the output of ls -la /booton a fresh installed bookworm?

pi@raspberrypi-bookworm64-lite-cm4:~ $ ls -la /boot
total 102028
drwxr-xr-x  4 root root     4096 Dec 28 14:36 .
drwxr-xr-x 19 root root     4096 Dec 29 15:44 ..
lrwxrwxrwx  1 root root       20 Oct 10 05:39 cmdline.txt -> firmware/cmdline.txt
-rw-r--r--  1 root root   230394 Oct 27 16:31 config-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root   230383 Oct 27 16:31 config-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root   230416 Nov 24 17:11 config-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root   230405 Nov 24 17:11 config-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       19 Oct 10 05:39 config.txt -> firmware/config.txt
-rw-r--r--  1 pi   pi         94 Dec 14 18:32 deletedKernels.txt
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware.bak
-rw-r--r--  1 root root 17125444 Dec 28 14:36 initrd.img-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root 17124794 Dec 28 14:36 initrd.img-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root 17126006 Dec 28 14:35 initrd.img-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root 17124187 Dec 28 14:35 initrd.img-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       17 Oct 10 05:38 overlays -> firmware/overlays
-rw-r--r--  1 pi   pi         94 Dec 16 23:40 raspiHandleKernels.krnl
-rw-r--r--  1 root root       83 Oct 27 16:31 System.map-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root       83 Oct 27 16:31 System.map-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-v8
-rw-r--r--  1 root root  8731342 Oct 27 16:31 vmlinuz-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root  8729756 Oct 27 16:31 vmlinuz-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root  8762266 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root  8760390 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-v8

[framps]

As you can see, it is NOT following to /mnt/clone/boot/firmware

In the echo command there is no realpath nor --follow-symlink used. In the corresponding sed command --follow-symlink ist used :wink:

If realpath is used the echo command will echo the correct filename and use the correct filename in sed.

So both ways work correctly but for the --symlink case the echo command echos not the final target filename.

[fmarzocca]

Aaaaaah! That /boot directory!

I fixed my symlinks in /boot. Now it works, thanks!

[fmarzocca]

@framps Have you reverted it back to realpath??? SHould I have to test everything again?

[jdrch]

Replaced SD. It is working, thanks.

SD cards fail in weird ways, often not all at once, either.

I think the reason @fmarzocca & @framps are seeing different things is there are a lot of underlying changes in the specific case of the Raspberry Pi OS v11.x to v12.x update that apt doesn't - and some that it can't - account for. I "successfully" updated a 3B+ from 11 to 12 myself, only to run into apt update problems later because packages, dependencies, etc. weren't located in the filesystem where Raspberry Pi OS v12 apt expected them to be. I wound up having to clean install to resolve the issues.

@fmarzocca I don't know the exact reason a clean install won't work for you, but in my case I'd avoided doing so due to an application, UniFi Controller, with dependencies that can't be fulfilled (not by default, at least) by later versions of Raspberry Pi OS. Fortunately, I found instructions online on how to do so with a clean install. You may wind up having to do something similar or - as is possible with UniFi Controller - migrate to a 1st party appliance that has already solved that problem.

[fmarzocca]

@fmarzocca I don't know the exact reason a clean install won't work for you,

I didn't say that it won't work, but that it is a major PITA. My server has a complex NodeRed installation (which can be easily backed up, but it also have several (maybe 7 or 8) additional services that have been carefully tuned up during years (i.e.: apcupsd, gammu, email, mosquitto and many others). It is not just a matter of backup the home folder. To do a complete new installation I should save all the settings and stop the server for several days (I don't have full time free) and this will completely stop the domotics. And I am sure I will forgot something. That's why I took the way of an upgrade which - I may be honest - is fully working for ten days without problems. I have fiuxed this last issue with rpi-clone.

[framps]

Have you reverted it back to realpath

Yes. That way the echo displays the symlinked filename.

So now the code uses realpath again with your suggested modification in line 1730 :wink:

[fmarzocca]

I "successfully" updated a 3B+ from 11 to 12 myself,

@jdrch 32-bit or 64-bit?

[jdrch]

I "successfully" updated a 3B+ from 11 to 12 myself,

@jdrch 32-bit or 64-bit?

32-bit.

All subsequent deployments have been 64-bit.

[phillipredshirt]

Following the link from this comment

"I think that's not dependent on the new HW but the SW required by RPi5. As far as I know RPi5 requires bookworm and rpi-clone works with bookworm if you apply the patch listed in https://github.com/billw2/rpi-clone/issues/168."

rpi-clone fails for me when trying on my RPI 5

What is the patch? It seems like this is still in flux.

[fmarzocca]

What is the patch? It seems like this is still in flux.

@phillipredshirt use this fork, it has been patched and working.

[phillipredshirt]

It's still not working. During boot I get:

`Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done Begin: Waiting for root file system ... Begin: Running /scripts/local-block ...done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done. Begin: Running /scripts/local-block ... done.

same line about 15 times then

done. Gave up waiting for root file system device. Common problems .

... ALERT! PARTUUID=20ffxxxx-02 does not exist. Dropping to a shell!

`

That's the PARTUUID of the SD card.

[phillipredshirt]

@fmarzocca I used the fork and it didn't change anything and then I used the -l switch on the command like this

sudo rpi-clone -l sda

and now I can't boot from the SD card either. It get the same error but points to the PARTUUID of the SSD which isn't plugged in.

So, I kept both the SD card and the SSD and it boots. I think my pi is a little mixed up now.

Edit:

Since it looked like "rpi-clone sda" was failing to update the PARTUUID information on the SSD and "rpi-clone -l sda" modified the SD to point to the PARTUUID of the SSD, I decided to run "rpi-clone sda" one more time. And it worked.

That was weird and a little scary. But my Pi 5 is now running on just my SSD.

Thank you, @fmarzocca

[matatata]

Thank you @framps !! It took me some digging to understand what was going wrong. Once I knew I also found this issue here and the fork which worked very well when cloning to usb. I finally could boot it, because the cmdline.txt was updated properly on the usb. Please @billw2 look into the fork or at least make a note in the readme that current bookworm systems have this problem.

[jdrch]

Thank you @framps !! It took me some digging to understand what was going wrong. Once I knew I also found this issue here and the fork which worked very well when cloning to usb. I finally could boot it, because the cmdline.txt was updated properly on the usb. Please @billw2 look into the fork or at least make a note in the readme that current bookworm systems have this problem.

Reasonably sure this repo has been abandoned.

[teq0]

Not working with Bookworm at all for me, even with the patch. When I do

sudo rpi-clone sda -v -f

when it boots it ends up saying

Gave up waiting for root file system device...

and sits in a Busybox shell with an initramfs prompt.

Using

sudo rpi-clone sda -v

after booting it ends up like

Trying partition: 0
lba: 8192 oem:  'mkfs.fat' volume: ' NO NAME   '
...
Firmware not found

Tried with 2 different SD cards. What am I doing wrong? This worked fine on another Pi running buster.

[teq0]

Pi 4. And it's the lite image.

Does Raspi Imager change the image based on the machine? I've burned a bunch of Bookworm Lite cards and told it it was for Pi 4 but they run fine in 3 and 3+s.

I'll try cloning Bookworm on a 3 and see what happens.

[jdrch]

I think you wind up with 2 different images if you update from 11.x vs. clean install 12.x.

[framps]

I may be wrong but I guess my fork of rpi-clone fails if Bookworm is not a fresh install but an upgrade from Bullseye. In #174 I helped the guy to check the UUIDs and they didn't match even rpi-clone updated /boot/cmdline.txt. But if /boot/cmdline.txt is a file and no link to /boot/firmware/cmdline.txt rpi-clone updates /boot/cmdline.txt but not /boot/firmware/cmdline.txt. But this file is not used during boot and therefore boot fails because of UUID mismatch.

Maybe somebody who runs an upgraded Bookworm can verify whether my guess is correct?

[jdrch]

fails if Bookworm is not a fresh install but an upgrade from Bullseye

This has been my experience too.

Maybe somebody who runs an upgraded Bookworm can verify whether my guess is correct?

The irony of that request is because the 11.x to 12.x upgrade path isn't officially supported, upgraded 12.x installations eventually run into breaking issues, such apt being unable to satisfy dependencies.

In other words, as horrible as it sounds, I don't think supporting upgraded 12.x installations is worthwhile, as it's pretty much impossible to guarantee the config/folder structure/locations the script is operating on.

[framps]

11.x to 12.x upgrade path isn't officially supported

That's why I install a fresh OS every time I want to upgrade my systems and I don't have a system handy which was upgraded to verify my guess.

Actually it's not a big deal to check whether /boot/cmdline.txt is a file and create a link to /boot/firmware/cmdline.txt. But I frankly don't like to fix this under the cover in rpi-clone. Maybe it's worth to check and write an error message and nobody blames rpi-clone .

[jdrch]

That's why I install a fresh OS every time I want to upgrade my systems and I don't have a system handy which was upgraded to verify my guess.

Requiring clean installs for major version upgrades is pretty retrograde and anachronistic of the Raspberry Pi Foundation (I'm not blaming @framps, @billw2, or anyone else) in 2024 AD when the vast majority of major OSes, including the one Raspberry Pi OS is based on, support in-place upgrades. Indeed, Debian itself supports 11.x to 12.x upgrades. I know this because I did it just fine with my Debian server.

I appreciate the Foundation warning users of the risks of in-place upgrades, but the fact they aren't supported reflects poorly on the Foundation and the Foundation only.

Maybe it's worth to check and write an error message and nobody blames rpi-clone .

Correct. This is a Raspberry Pi OS, and therefore a Raspberry Pi Foundation problem. Except they don't see it as a problem.

I'd say rpi-clone should probably have an option to toggle between cloning for the following environments:

1. v12.x, clean installed
2. all versions up to 11.x

As I said in a previous comment, I don't think it's fair to developers to ask them to support 11.x to higher major version in-place upgrades if the underlying OS itself doesn't support them.

[fmarzocca]

I have upgraded 4 Raspi 3B+ from bullseye to bookworm, and I am happily using @framps' fork, provided to execute (after kernel update) the following:

cd /boot
sudo ln -s firmware/cmdline.txt ./cmdline.txt
sudo ln -s firmware/config.txt ./config.txt

[framps]

This proves an upgraded OS misses the links which are new to Bookworm and everybody who upgraded has to execute these commands first and then rpi-clone will create a bootable clone.

[fmarzocca]

This proves an upgraded OS misses the links which are new to Bookworm and everybody who upgraded has to execute these commands first and then rpi-clone will create a bootable clone.

Correct.

[ProblemChild4901]

I am suprised this this works. I would have thought that a bullseye to bookworm upgrade would retain the old /boot mountpount, creating the /boot/firmware directory within the /boot filesystem. I was under the impression that vfat filesystems do not support unix symbolic links.

[framps]

Keep in mind the boot partition is mounted on Bookworm now at /boot/firmware and /boot is located on the root filesystem which is formatted with ext4 and supports symlinks.

pi@raspberrypi-bookworm64-lite:~ $ ls -la /boot
total 39292
drwxr-xr-x  3 root root     4096 Jan  8 21:33 .
drwxr-xr-x 18 root root     4096 Dec 11 05:55 ..
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       20 Dec 11 05:37 cmdline.txt -> firmware/cmdline.txt
-rw-r--r--  1 root root   230416 Nov 24 17:11 config-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root   230405 Nov 24 17:11 config-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       19 Dec 11 05:37 config.txt -> firmware/config.txt
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware
-rw-r--r--  1 root root 11108528 Dec 11 05:54 initrd.img-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root 11108073 Dec 11 05:54 initrd.img-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       18 Dec 11 05:54 issue.txt -> firmware/issue.txt
lrwxrwxrwx  1 root root       17 Dec 11 05:36 overlays -> firmware/overlays
-rw-r--r--  1 root root  8762266 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root  8760390 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-v8

pi@raspberrypi-bookworm64-lite:~ $ mount | grep mmcblk0p
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
/dev/mmcblk0p1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

[ProblemChild4901]

I see unofficial rpios upgrade bullseye to bookworm instructions that do not mention upgrading to the new kernel or say that is is optional. Upgrading the kernel will remount /boot to /boot/firmware like a new installation of bookworm, but likely will not add the symbolic links. I just upgraded a 32bit lite rpios using the "edit apt sources" method. While the upgrade to bookworm works, I was left with this:

pi@raspberrypi:/boot $ mount |grep mmcblk0 /dev/mmcblk0p2 on / type ext4 (rw,noatime) /dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

pi@raspberrypi:/boot $ df -h / /boot Filesystem Size Used Avail Use% Mounted on /dev/root 59G 2.1G 54G 4% / /dev/mmcblk0p1 255M 51M 205M 20% /boot

pi@raspberrypi:/etc $ cat os-release |head -1 PRETTY_NAME="Raspbian GNU/Linux 12 (bookworm)"

pi@raspberrypi:/etc $ ls -la /boot/config.txt /boot/cmdline.txt -rwxr-xr-x 1 root root 102 Dec 4 21:57 /boot/cmdline.txt -rwxr-xr-x 1 root root 2075 Dec 4 21:39 /boot/config.txt

In this specific case, the symbolic links not only are not present, but probably are not requred for rpi-clone to work.

[framps]

Bookworm uses /boot/firmware/cmdline.txt during boot and if you modify /boot/cmdline.txt which is no link to /boot/firmware/cmdline.txt this update will be ignored.

That's the root cause for this issue because rpi-clone fails to update the UUID in /boot/firmware/cmdline.txt but instead updates /boot/cmdline.txt.

[jdrch]

that do not mention upgrading to the new kernel or say that is is optional

The fact that's the case tells you how broken the in-place upgrade path is.

[geerlingguy]

@framps thanks for maintaining your fork / patchset! I've also pulled it into my fork, so hopefully anyone looking for a working tool can grab one of the forks (I had been maintaining my fork for the NVMe support, so since it's referred to a lot I decided to pull in your changes instead of just archiving it... would love to find a way for there to be one 'official' place for the project to reside!

Alternatively, does your rpiBackup tool potentially have room for cloning functionality built in? (Last time I looked, I think you had to do some sort of backup + restore to another drive, which was more complex than a straight clone).

[framps]

so since it's referred to a lot I decided to pull in your changes instead of just archiving it...

NP. Glad you pulled my changes. Whoever uses rpi-clone should be able to use rpi-clone with Bookworm and future OS releases. Whether it's grabbed from your fork or mine :smiley:

would love to find a way for there to be one 'official' place for the project to reside!

Agree. Given the fact I maintain raspiBackup I don't volunteer to maintain rpi-clone - even I was asked to do this already.

Alternatively, does your rpiBackup tool potentially have room for cloning functionality built in? (

raspiBackup was designed to create backups - not to clone any systems. Indeed you can clone a system by creating a backup and restore this backup. But that's a workaround and not convenient.

I frankly think most of the guys actually want to create a backup instead of a clone. Maybe I'm wrong ... If there is a strong requirement to get a cloning capability in raspiBackup I'll think about this new feature.

[geerlingguy]

@framps heh, I want both! :D

But for me, the cloning is often useful just to test different media on the same Pi without having to re-image (though sometimes imaging fresh is faster anyway).

For most people, I think full backups are all they'd ever need, short of a better mechanism to upgrade major versions (something I still never recommend on any Linux OS!).

[framps]

the cloning is often useful just to test different media on the same Pi

I see your point but that's a special use case you have :smiley:

Most of the folks using raspiBackup just want to have a backup in case either the SD card fails or something went wrong when they do any upgrades or manually do any config updates and which causes the system fail to boot.