Support UUID= (filesystem UUIDs), Armbian and partially Ubuntu

[matthijskooijman]

This makes two changes:

1. Supporting systems (such as Armbian) that use UUID= (filesystem uuids) rather than PARTUUID= (partition uuids)
2. Suppporting Armbian by editing /boot/armbianEnv.txt instead of /boot/cmdline.txt if the former exists but the latter does not.

With these changes, I was able to run rpi-clone on an Orange Pi PC running Armbian (but I expect it to run on most if not all other Armbian-supported boards as well). I tried both a stock image with a single partition, as well as a self-compiled image with 2 partitions (/boot and /). I also tried imaging SD to USB with -l, which then boots from USB succesfully, and then swapping out the SD card and image back from USB to SD (without -l), which produces a working SD again.

One thing I did not implement is changing the FS UUID for partitions that are copied using dd, so for these partitions you might end up with duplicate UUIDS. I'm not exactly sure when this happens (I think only for partitions that are not mounted, so maybe the duplicate UUIDs are not problematic directly), but this might need a better looking at.

Note that this PR includes the commits of #101, since I needed both so I developed this on top of that and because they touch nearby code, there's merge conflicts when trying to untangle them (which is possible, but would only result in more conflicts when trying to merge both, so I thought it better to just leave them entangled). Effectively this means only the last three commits should be reviewed here, the first 5 are subject of #101.

[matthijskooijman]

I just pushed one more commit that enables modifying /boot/firmware/cmdline.txt, which where the cmdline is stored in Ubuntu installations. If you modify an Ubuntu installation to use (PART)UUID instead of filesystem labels, this should allow cloning the installation.

@ioogithub, Could you test this?

[ioogithub]

To confirm, can I test this PR with the following:

git clone https://github.com/matthijskooijman/rpi-clone.git
cd rpi-clone
git checkout support-fsuuid-and-armbian
sudo ./rpi-clone device

[matthijskooijman]

Yup, that sounds like the correct approach.

[ioogithub]

I have tested the code, at first it appeared that it had worked because I got this result at the end of the clone:

Editing /boot/firmware/cmdline.txt PARTUUID to use abcdef12
Editing /mnt/clone/etc/fstab PARTUUID to use abcdef12
===============================
Done with clone to /dev/sdb

however when I inspected /mnt/clone/boot/firmware/cmdine.txt I saw this that the PARTUUID was the same on the destination sd card as the original. So even though it said reported that it edited the file it did not or perhaps it did change it but another part of the code changed it back?

I ran the test again with the modified code (above) and I confirmed that it did work.

Im not sure why this change did not work, from looking at your code it looks like it should and it even reported that it made the change so perhaps there is something else that overwrites it back afterwards?

[matthijskooijman]

however when I inspected /mnt/clone/boot/firmware/cmdine.txt I saw this that the PARTUUID was the same on the destination sd card as the original.

Hm, weird. Did you check that the actual PARTUUIDs were different (or alternatively, that the PARTUUID show in the log output really did not match cmdline.txt)?

I ran the test again with the modified code (above) and I confirmed that it did work.

You mean you ran the test a second time without changing anything? Or is your "above" referring to the other issue where you posted a manual change of the cmdline.txt file?

[ioogithub]

Hm, weird. Did you check that the actual PARTUUIDs were different (or alternatively, that the PARTUUID show in the log output really did not match cmdline.txt)? I checked /boot/firmware/cmdline.txt immediately after the operation and noticed that it had the PARTUUID from the source card.

I did notice another unusual event, several hours later I noticed that after a restart I was booted from the /dev/sdb which was was completely unexpected. Is it possible that the code changed the cmdline.txt on the source card and not the destination?

I will do another test, before the test I will check both fstab and cmdline.txt then I will check then again after and report back.

[matthijskooijman]

Is it possible that the code changed the cmdline.txt on the source card and not the destination?

Yup, that is exactly what happened, thanks for catching that. Turns out the UUID version prepends ${clone} as expected, but the PARTUUID version does not. I thought I had tested this change on Raspbian as well, but maybe I just didn't notice that it booted from the wrong card or something.

I'll push a correction later, maybe tomorrow.

[diagonali]

Is it possible that the code changed the cmdline.txt on the source card and not the destination?

Yup, that is exactly what happened, thanks for catching that. Turns out the UUID version prepends ${clone} as expected, but the PARTUUID version does not. I thought I had tested this change on Raspbian as well, but maybe I just didn't notice that it booted from the wrong card or something.

I'll push a correction later, maybe tomorrow.

I've spent a lot of time wrestling with trying to get this super useful script which used to work easily with Raspberry Pi OS to use instead with Ubuntu 22.04 on a Raspberry Pi 4 (nothing I tried would get it to boot after cloning with the original script), I finally found your branch and all is now working as expected.

I can clone from sda to mmcblk0 and vice versa and its super useful to be able to keep an SD card in the Pi at all times with a working, bootable copy of the system so that in the event the SSD is unusable or unbootable, I can simply remove the SSD and the Pi will boot from SD where I can then clone back to USB and carry on as before. Thanks so much for your efforts, they're certainly appreciated here! I do hope this script can continue to be improved and I'm also surprised there aren't more similar scripts out there as it's functionality is so useful.

[matthijskooijman]

Yup, that is exactly what happened, thanks for catching that. Turns out the UUID version prepends ${clone} as expected, but the PARTUUID version does not. I thought I had tested this change on Raspbian as well, but maybe I just didn't notice that it booted from the wrong card or something.

I've just pushed a fixup commit to fix this issue. I haven't tested it, @ioogithub maybe you can check if it works for you now?

@diagonali, good to hear this branch has been useful for you. I'm assuming you have change fstab and cmdline.txt from labels to (PART)UUIDs manually before applying this branch? Also, did you run into the issue from https://github.com/billw2/rpi-clone/pull/140#issuecomment-1108779789? Or did you maybe use UUID instead of PARTUUID (UUID already worked, PARTUUID was broken until just now).

[diagonali]

@matthijskooijman To be honest, despite tinkering with linux for a quite a while now, I haven't fully understood fstab, UUIDs etc and how they relate to use in this script so I didn't do any manual changes at all. Best I tried was specify a label using the original script to try to get the cloned drive to boot in Ubuntu but it didn't work. I'll definitely need to learn about these technicalities eventually though!

I simply installed a fresh Ubuntu 22.04 onto a Raspberry Pi 4 using USB boot and ran your branch of this script to clone to an SD card. Then I removed the USB drive and rebooted and the system boots successfully. Then I plugged in the USB drive and cloned back to that and removed the SD card and again it boots successfully. As far as I can tell there is some new messaging (relating to labels I think?) during runtime which I assume is due to your edits. Also I noticed that running the script with -f flag doesn't seem to do the same thing(s) as running without this flag in terms of making the cloned drive bootable? I didn't get the same issue as the #140 comment as far as I can see.

I've just pushed a fixup commit to fix this issue. I haven't tested it, @ioogithub maybe you can check if it works for you now?

I'm using the following commands to use your branch - use this again to update?:

git clone https://github.com/matthijskooijman/rpi-clone.git
cd rpi-clone
git checkout support-fsuuid-and-armbian
sudo ./rpi-clone device

[matthijskooijman]

I simply installed a fresh Ubuntu 22.04 onto a Raspberry Pi 4 using USB boot and ran your branch of this script to clone to an SD card. Then I removed the USB drive and rebooted and the system boots successfully.

Ah, you're only keeping either the USB drive or SD card plugged in, that should work even without manual fstab changes indeed (if you plug in both, it might be that you're booting one, but end up with the root filesystem on the other because the labels are identical). This also means that you do not really need this PR (updating UUIDs in cmdline.txt), but #101 is sufficient (but that is included in this PR). if you would convert fstab and cmdline.txt to UUIDs because you want to be able to have both plugged in at the same time, then you would need this PR.

I'm using the following commands to use your branch - use this again to update?:

You could probably also get away with a simple git pull in the existing clone (because I did not do a force push), but using a fresh clone as you suggest will also work fine (you might need to delete the existing clone first).

[diagonali]

if you would convert fstab and cmdline.txt to UUIDs because you want to be able to have both plugged in at the same time, then you would need this PR.

@matthijskooijman Interesting, I definitely want to be able to keep the SD card inserted when using USB boot (and vice versa) as the SD card operates in my case as a "live" bootable failsafe "backup" in case of issues with the USB drive/installation. It makes sense from what you say that it could boot one but use the filesystem of the other - not ideal obviously!

Question for me now is how to as you say "convert fstab and cmdline.txt to UUIDs". I'll start looking into (Googling) how to do this in Ubuntu on the Raspberry Pi and I'm sure there are guides online. I'm wondering if you could give any simple instructions if this is a simple process? I'll be doing this on a live system so want to make sure I do this correctly. I use my Raspberry Pi as a home automation server so the setup I have needs to be as robust as possible and I avoid the boot/filesystem overlap you mention.

Also I'm wondering if this process could be incorporated into the script itself? Is switching over to using UUIDs too tricky to process in an automated way?

[matthijskooijman]

@diagonali See the discussion starting at https://github.com/billw2/rpi-clone/issues/100#issuecomment-865924895

Also I'm wondering if this process could be incorporated into the script itself? Is switching over to using UUIDs too tricky to process in an automated way?

It could probably be done (there is already an option and code to switch from device names (e.g. sda1) to PARTUUID), but I won't be doing this (This PR has already gotten more extra changes beyond what I need myself).

[diagonali]

@diagonali See the discussion starting at #100 (comment)

Also I'm wondering if this process could be incorporated into the script itself? Is switching over to using UUIDs too tricky to process in an automated way?

It could probably be done (there is already an option and code to switch from device names (e.g. sda1) to PARTUUID), but I won't be doing this (This PR has already gotten more extra changes beyond what I need myself).

Ah ok no problem, thanks very much for what you've done so far.

[matthijskooijman]

I did a bit more work on this PR to make rpi-clone update UUIDs in non-mounted partitions as well (plus some refactoring leading up to that). I needed this to work with a dual-boot image/SD-card I have been working with (single image that contains boot+root for both a raspberry pi and an orange pi), but it might be useful for other cases as well.

I did a force push to squash the fixup commit I previously pushed.

Edit: And one more commit to fix permission warnings on FAT filesystems mounted with the uid option, which is really unrelated but I just wanted to publish it somewhere without adding yet another PR

[fjpdevries]

Ik ben tot en met vrijdag 5 augustus 2022 afwezig en zal mijn email niet of beperkt kunnen lezen en beantwoorden. Neem voor dringende zaken contact op met mijn collega André de Jonge, email @.***, tel. 0224-297819.

Met vriendelijke groet, FSE Turnstiles b.v. Jos de Vries

[KcrPL]

I know it's been some time since someone commented on this but I'd really like to thank you. It took me a lot of time this evening and your fork was a solution to this. Thank you :)

[timo12357]

I have an Orangepi 3LTS running Armbian 23.8.1 Bullseye with Linux 5.15.93-sunxi64. sys-clone works perfect as long as I have the system on a SD card. I moved the system to the EMMC. The initial sync works, but sys-clone fails to do a subsequent sync complaining that it can not mount the clone disk:

`sudo sys-clone sdc

Destination disk partition /dev/sdc1 is mounted on /mnt/clone. The clone cannot proceed unless it is unmounted. Do you want to unmount /mnt/clone? (yes/no): yes

Booted disk: mmcblk2 7,8GB Destination disk: sdc 31,3GB

Part Size FS Label Part Size FS Label 1 root 7 2G ext4 -- 1 29 1G ext4 --

== SYNC mmcblk2 file systems to sdc == / (2,9G used) : SYNC to sdc1 (29,1G size)

Run setup script : no. Verbose mode : no. -----------------------:

Ok to proceed with the clone? (yes/no): yes

Syncing file systems (can take a long time) Syncing mounted partitions: e2label /dev/sdc1 -- Mounting /dev/sdc1 on /mnt/clone mount: /mnt/clone: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error. Mount failure of /dev/sdc1 on /mnt/clone. Aborting!`

I have tested this with two separate SD cards, and same error occurs. Is this a bug or user error?

[matthijskooijman]

Hm, so it fails to mount /dev/sdc1 saying the FS is invalid (my interpretation), but it was mountd when you started. Maybe the script messed up the filesystem trying to copy something? Can you manually mount /dev/sdc1 before and after running the script?

Btw, I think this might be unrelated to this particular pullrequest - Yes, you're using Armbian, but the changes in this PR are mostly about updating filesystem references for Armbian, and your problem seems to occur way earlier before even starting to copy files. So maybe it would be better to open up a separate issue for this? Maybe comment with a link to that issue here?

[timo12357]

This still does not work after updating to Armbian 23.8.3. Anyone else seeing this behavior, or is this a problem with my setup?