Could not stat device /dev/mmcblk0 on a CM4

[framps]

I just wanted to evaluate your script and tried to create a backup and immediately got following error message:

## Zoom speed NOT requested...
## Scanning filesystem and calculating...
Error: Could not stat device /dev/mmcblk0
/dev/mmcblk0boot0
/dev/mmcblk0boot1 - No such file or directory.

!! Script stopped by user...
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 00.10

The script just hangs and I had to cancel the script. It also doesn't produce any log file I can provide for further analysis of the issue.

It's a cm4 with following partitions:

pi@raspberrypi:~ $ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0  29.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0  14.8G  0 part /
mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 
nvme0n1      259:0    0 119.2G  0 disk 
├─nvme0n1p1  259:1    0   512M  0 part 
└─nvme0n1p2  259:2    0 118.7G  0 part 

[framps]

The cm4 is dual boot capable and I booted the OS from nvme instead of mmcblk and now get

pi@raspberrypi:~ $ sudo ./shrink-backup -a -l /backup/shrinkBackup.img
## Debugging requested, writing to log file: ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
lsblk: 

/dev/nvme0n1p1: not a block device
#####################################################################################
# A backup will be created at /backup/shrinkBackup.img                              #
# ext4 filesystem detected on root                                                  #
# --------------------------------------------------------------------------------- #
# Write to logfile: true                                                            #
# Zoom speed requested: false                                                       #
# Autocalculate img root partition size: true                                       #
# Autoexpand filesystem at boot: true                                               #
# Use exclude.txt: false                                                            #
# Bootsector size: 515MiB                                                           #
# Estemated root usage: 7328MiB                                                     #
# Auto calculated size (root partition): 9139MiB                                    #
# Total img size: 9655MiB                                                           #
#####################################################################################
## Do you want to continue? [y/n] y

## Creating bootsector...

530969088 bytes (531 MB, 506 MiB) copied, 8 s, 66.4 MB/s
1067007+0 records in
1067007+0 records out
546307584 bytes (546 MB, 521 MiB) copied, 8.72056 s, 62.6 MB/s
## Resizing img file...
## Looping img file...
!! LOOP paths can not be set, retrying in 5 seconds...
!! LOOP paths can not be set, retrying in 5 seconds...
!! LOOP paths can not be set, retrying in 5 seconds...
!! LOOP PATHS CAN NOT BE SET!!!
!! Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 00.46

I now got a debug log I attached :-)

[framps]

shrink-backup.log

[UnconnectedBedna]

Hi and thank you for making a bug report.

Since I am using uuid to find correct partitions, can you please provide the output of lsblk -o name,fstype,label,uuid,partuuid,ptuuid

Can you also shed some light on what these are?

mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 

I have not even thought about dualbooting, but if you are willing to help me out a bit we might be able to figure this out. It should be doable, but just so you are aware/understand, it will ONLY backup one or the other (mmcblk OR nvme depending what you have booted into) to the img file, not all partitions.

It would be really helpful to read a log file for the failing mmcblk. Or do you mean you started the script using -l and it did not create a log entry into the logfile?

Edit It must fail with this on line 2030: mount | grep ' / ' | awk '{print $5}' Or some of the following lsblk lines because it complains about it on your nvme backup too:

lsblk: 

/dev/nvme0n1p1: not a block device

Typing following in a console while booted using mmcblk, what does that return for you?

mount | grep '  /  ' | awk '{print $5}'
lsblk -lpo mountpoint,ptuuid | grep '/ ' | awk '{print $2}' # this goes in the line below
lsblk -lpo ptuuid,type,path | grep <insert the ptuuid from line above> | grep 'disk' | awk '{print $3}'

End edit

---

As for the second backup with the log, it fails here:

2024-07-21 22:40:50 [DEBUG] - LOCAL_DEV_BOOT_PATH=

/dev/nvme0n1p1 | LOCAL_DEV_ROOT_PATH=/dev/nvme0n1p2
2024-07-21 22:40:50 [DEBUG] - LOCAL_BOOT_UUID= | LOCAL_ROOT_UUID=fc7a1f9e-4967-4f41-a1f5-1b5927e6c5f9 | LOCAL_ROOT_PARTUUID=60b7d56f-02

It does not find the path or uuid of boot.

When booted using nvme, what is the output of lsblk -lpo mountpoint,path | grep 'boot' | awk '{print $2}' and lsblk -o uuid /dev/nvme0n1p1 | tail -1

Edit I was a bit tired writing that, ofc it's not going to work, it also finds those 2 partitions disks I asked about in the beginning of the post, so we have to figure out a way to do this without using ONLY the word "boot". xD Maybe only looking for parts instead of also including disks...

If you feel up to the task, edit shrink-backup on line 354:

- LOCAL_DEV_BOOT_PATH=$(lsblk -lpo mountpoint,path | grep 'boot' | awk '{print $2}')
+ LOCAL_DEV_BOOT_PATH=$(lsblk -lpo mountpoint,path,type | grep 'part' | grep 'boot' | awk '{print $2}')

And retry. End edit

---

There is another user reporting his boot unmounts by itself (or he thinks it might be connected to the script somehow, either after pressing ctrl+c or pressing "n" on the question to continue), please make sure this is not something that is occurring for you, because it would show this exact behavior.

[framps]

pi@raspberrypi:~ $ lsblk -o name,fstype,label,uuid,partuuid,ptuuid
NAME         FSTYPE LABEL  UUID                                 PARTUUID                             PTUUID
mmcblk0                                                                                              d5c63721
├─mmcblk0p1  vfat   bootfs 44FC-6CF2                            d5c63721-01                          d5c63721
└─mmcblk0p2  ext4   rootfs 93c89e92-8f2e-4522-ad32-68faed883d2f d5c63721-02                          d5c63721
mmcblk0boot0                                                                                         d5c63721
mmcblk0boot1                                                                                         d5c63721
nvme0n1                                                                                              60b7d56f
├─nvme0n1p1  vfat   bootfs 50C8-AEAE                            60b7d56f-01                          60b7d56f
└─nvme0n1p2  ext4   rootfs fc7a1f9e-4967-4f41-a1f5-1b5927e6c5f9 60b7d56f-02                          60b7d56f

have not even thought about dualbooting,

I don't think it makes sense to handle this. It's sufficient to backup the active system. I have setup this dual boot on my CM4 just to easily switch between two OSes.

Typing following in a console while booted using mmcblk, what does that return for you?

mount | grep '  /  ' | awk '{print $5}'

returns nothing :-(

pi@raspberrypi:~ $ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=194008k,nr_inodes=48502,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=92896k,mode=755)
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2368)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/mmcblk0p1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=92892k,nr_inodes=23223,mode=700,uid=1000,gid=1000)

If you feel up to the task, edit shrink-backup on line 354:

When I updated the line the backup image was created successfully. I now have to test the restore.

I unfortunately didn't create a second issue for the nvme booted system issue. Looks like this issue is fixed with the updated line (restore verification pending ...).

Can you also shed some light on what these are?

I frankly don't know. But I think it's something special you have on a CM4 for eMMC storage.

[UnconnectedBedna]

Please let me know if the restore works. :smiley:

It's really odd that the line returns nothing because grep should find: /dev/mmcblk0p2 on / type ext4 (rw,noatime) And awk should filter out the fifth collumn, witch is ext4 Maybe you don't have grep or awk installed? But that should return an error at least...

Does this line work? cat /proc/mounts | grep ' / ' | awk '{print $3}'

---

I frankly don't know. But I think it's something special you have on a CM4 for eMMC storage.

Can you boot the device without the sd-card inserted? (I'm just curious here)

If you verify that this solves the issue for you I will implement it in the next update. If you want to push the change yourself I will accept the pr, but do it to the testing branch in that case (I saw the fork. xD) If you do, please also reference this issue (#40) in the pr. No pressure, it's just a question if you want to... :slightly_smiling_face:

[framps]

Please let me know if the restore works.

Sure. Looks like I don't have a 16GB SD card handy to test the restore :-( Will keep you posted on this.

Therefore I just test a backup and restore of a RPi3 running from a 8GB SD card right now.

Maybe you don't have grep or awk installed? But that should return an error at least...

Then I will get an error awk or grep not found ;-)

Does this line work? cat /proc/mounts | grep ' / ' | awk '{print $3}'

No, but cat /proc/mounts | grep ' / ' | awk '{print $3} returns ext4.

Also mount | grep ' / ' | awk '{print $5}' returns ext4.

Note you used two leading and trailing spaces. It works with one leading and trailing space.

Can you boot the device without the sd-card inserted? (I'm just curious here)

I can't. It's a CM4 with on board eMMC storage.

If you verify that this solves the issue for you I will implement it in the next update.

Not sure which change you're talking about. Unfortunately we talk about multiple issues in one issue now :-(.

Just an update: I just started the restored image of my RPi3 and the boot fails with a lot of messages

Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device. 
...
ALERT! PARTUUID=a1f2b077-02 does not exist. Dropping to a shell!

So now we have 3 issues in one issue :-(((

I'll check now the PARTUUIDs in the image. I'll create another issue for the RPi3 restore boot failure when I have checked the backup image.

[UnconnectedBedna]

The first 2 issues are probably the same, don't worry. But I question if you can just switch over a nvme to sd or mmcblk like that, but I am very curious as to how it works. sd to mmcblk and vice versa on rpi works and have been tested by me, but nvme and cm4 could be different so please let me know, I don't own that kind of hardware.

And the leading and trailing space was always there, otherwise you should get every single mount with the line, not nothing. The line you quote and the line you say work are literally the same. xD But I did notice github for some reason removed the spaces when I used ctrl+v so I had to manually add them in again, maybe it's the same if you press the "copy code" button. :wink: Edit Now also more consistent in the actual code on the testing branch after this conversation https://github.com/UnconnectedBedna/shrink-backup/commit/7a1302f930c55b0e81ce3cd2ef1a16ff327dbed1 End edit

The error with your rpi3 is strange, I do not change uuid or partuuid, I don't change fstab and I don't change cmdline.txt but I do know that the rpi resize script in raspi-config does. So if you used that, made a backup with the script without first rebooting, yeah, this might happen... I also DO transfer the uuid from the root to img file when formatting the img filesystem, so something could go wrong there maybe but it's unlikely because that is the UUID not the PARTUUID. The 02 in PARTUUID=a1f2b077-02 tells me it's the second partition, ie root. But without a logfile (when you created the img) there is not much I can say. If it's a rpi3 32-bit, that could also be a reason, I do not own any 32 bit rpi (I think, now I have to go look) I can test the functionality on.

Edit: FYI, the error is in cmdline.txt on the boot partition. You can connect the sd card you wrote the img file to on another system and run lsblk -o name,partuuid and compare with the line in cmdline.txt. Then also check fstab and maybe you can figure out where the switch happened somehow. But a log file would be awesome. It does not make sense (unless you did the autoexpansion with raspi-config mentioned above), because the script dd:s the boot partition (ie it is a clone with the exact same PARTUUID), and that is what defines the PARTUUID for all other partitions on that device, trailing partitions just changes the last digits, in this case 02 because it is the second partition.

[framps]

Do you want to support CM4 with eMMC with shrink-backup? Then let me know how I can help you to debug the issue. Otherwise I suggest to document CM4 is not supported by shrink-backup :wink:

[framps]

FYI: I just created a backup of my NVME OS on my CM4 with the patch in line 354 and the PARTUUIDs are now OK in the image and in cmdline.txt and /etc/fstab. I didn't test a restore because this will be a complicated task I don't want to execute. But it looks good so far.

It seems to me shrink-backup fails because of

mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 

which exist if an internal eMMC is used.

[UnconnectedBedna]

Do you want to support CM4 with eMMC with shrink-backup? Then let me know how I can help you to debug the issue. Otherwise I suggest to document CM4 is not supported by shrink-backup 😉

I would very much like that! :smile: I have actually been very close to buying one, just for the sake of making sure basic installs works on them, and maybe nvme, m2 hats and such works as well, but other things always gets in the way.

It seems to me shrink-backup fails because of

mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 

which exist if an internal eMMC is used.

I have made 2 commits to the testing branch that could be interesting for you to try out. The script no longer uses lsblk to identify the path. Pull latest and try out testing branch.

git pull
git checkout testing
chmod +x shrink-backup
sudo shrink-backup yadayadayada

[framps]

I grabbed your testing branch but I still get

pi@raspberrypi-bookworm-64-lite-cm4-emmc:~ $ sudo ./shrink-backup -al /backup/shrinkBackupcm4emmc.img
## Debugging requested, writing to log file: ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
Error: Could not stat device /dev/mmcblk0
/dev/mmcblk0boot0
/dev/mmcblk0boot1 - No such file or directory.

and no debug log.

[framps]

I invoked shrink-backup with bash -x. Maybe this helps to locate the root cause for the failure.

shrink-backup.log

[UnconnectedBedna]

and no debug log.

Makes absolutely no sense. Even the log you provided with bash -x says so multiple times. The first that is set to false is when I define the variables it sets the default value with is false but then later:

++ getopt -o Uatyelzh --long help,fix,f2fs,version,loop: -- -al /backup/shrinkBackup64CM4EMMC.img
+ options=' -a -l -- '\''/backup/shrinkBackup64CM4EMMC.img'\'''
+ '[' 0 -eq 0 ']'
+ eval set -- ' -a -l -- '\''/backup/shrinkBackup64CM4EMMC.img'\'''
++ set -- -a -l -- /backup/shrinkBackup64CM4EMMC.img
+ true
+ DEBUG=true

<skipping unnecessary info>

+ echo -e '\033[0;37m## \033[0;97mDebugging requested, writing to log file: \033[0;92m./shrink-backup.log'
## Debugging requested, writing to log file: ./shrink-backup.log
+ debug INFO 'Debugging requested, writing to log file ./shrink-backup.log'
+ local log_level=INFO
+ local 'log_message=Debugging requested, writing to log file ./shrink-backup.log'
+ '[' true = true ']'

etc etc etc

---

But, it's those zombie "disks" that are not really disks, but part of the mmcblk. Parted detects them and does not like them on line 2041 on the testing branch version: PARTITION_TABLE=$(parted "$LOCAL_DEV_PATH" print | grep -i 'Partition Table' | awk '{print $3}') And if it fails there (I honestly thought it was the lsblk command, but it is not) it will also fail when removing and recreating the root partition, pretty sure about that.

I was hoping it was just the lsblk, but it is parted that is giving the script a hard time. You can see it with the lsblk I requested from you earlier, they all have the same PTUUID (not to confuse with PARTUUID)

Since the script relies on dd:ing the boot partition, therefore also including the partition table with those strange "disks" it will never most likely not work and I would have to own a setup like that to really bite my teeth into it to stand any chance of getting that to work.

My guess is, since they are 4MiB that these are MBR:s somehow that makes it possible to boot from something other than mmcblk. The cm4 does apparently not work like the rpi4 where it is built into the emmc (or whatever it's called, the firmware) to look for boot:s on all interfaces until it finds one.

So don't hold your breath, if it will ever be supported. Without those "disks", the script works flawlessly.

Another wontfix, sorry.

And should I assume this comment is also true for this issue as well? https://github.com/UnconnectedBedna/shrink-backup/issues/41#issuecomment-2246307200

I leave this issue open for a day or two, but your reaction doesn't exactly make me want to do anything about this either since trying to help is not really appreciated on your side unless I bend you your will like a slave. :(

This is an open source project, do I really have to remind people this is done all for free by my good will... sigh.. :(

[framps]

I think we both have some misunderstandings we should clean up :wink:

I appreciate your immediate accepted the issue and replied. I have no clue why you think I see you as my slave who must fix the issue :cry:

I see it's disappointing to see somebody who evaluates your tool to create immediately two issues. I unfortunately used my CM4 first (I though it's just a different RPi4) but missed there are some major differences and shrink-backup is not tested with a CM4 with embedded eMMC. When I noticed this I thought I maybe can help you to get this fixed if it's a low hanging fruit. As far as I understand now it's not a low hanging fruit. So just close this issue with unsupported and document somewhere shrink-backup does not support CM4 and everything is fine :smile: .

When I got the feature request for NMVe support for raspiBackup the requestor from US fortunately donated a CM4 which allowed me to add and test NVMe support (At that time this was the only RPi which was able to use native NMVe storage). As far as I can see shrink-backup has no issue with NVMe (see here). That's important because a lot of folks now use NVMe storage on RPi5. Not sure whether you already tested shrink-backup on a RPi5 which uses NVMe storage.

I was upset yesterday because of your harsh replies I thought you don't appreciate the effort I spent to help you to locate the root cause of the issue. I spent multiple hours to repeat the backup/restore process and provided logs and even bash -x logs.

I also maintain multiple open source projects in my spare time. I'm also glad to get feedback with issues like you and help to locate the root cause of any bugs. That's why I spent my time to help you to locate the root cause of the issue. But again - I got the impression you don't appreciate this effort.

I'm used to create issues whenever I find some just to help to improve any open source project. Whether this issue is accepted and fixed or rejected the project owner has to decide. And it also happens issues are created which are invalid because they are not tested and supported and will be closed silently as unsupported.

I leave this issue open for a day or two, but your reaction doesn't exactly make me want to do anything about this either since trying to help is not really appreciated on your side unless I bend you your will like a slave. :(

As I wrote you're not my slave and I don't force you to add support for CM4 eMMC. I think it's unfortunately just a misunderstanding :cry:

Final note: When I finally was able to backup and restore an image with shrink-backup I advertised your script in the German Raspberry forum as another nice backup tool for Raspberries.

And should I assume this comment is also true for this issue as well?

No. It's neither true for the other issue nor this one. It is always a good strategy to sleep on a matter :smile:

[UnconnectedBedna]

I could def have used better phrasing, I REALLY appreciate your and anybody's feedback! Feedback is what helps me advance the script. :) I am sorry about that!

I have spent some time thinkgin about the cm4, and came to this conclusion. It is most likely possible to get it to work even with the "boot disks" on mmcblk, BUT it involves quite a bit of trickery. For nvme, the backup should go through, but how it will work writing that img to an sd-card and boot, I have no idea, but writing it back to the nvme, and keeping the sd-card in, should work aswell, but that might not be what you are looking for?

The problem with backing up the mmcblk is that the "boot disks" are AFTER the root partition, bot before. If they were either before or directly after the boot partition, I actually think the script would work. But again, I have no idea how the boot process works on a cm4. :(

I am VERY willing to work with you to try to fix this, but it involves pretty deep knowledge of how the script works, and again, I CAN NOT TEST these things, I would have to rely on you 100%, and it feels bad from my side, not because I don't trust you, but because I put a lot of pressure on you...

Again, I REALLY APPRECIATE YOUR HELP/FEEDBACK!!!

If you want to open the other issue again, I will, but tbh, I think focusing on this would be way more interesting and useful. But it is up to you since the same is true there, I have no way of confirming everything works 100%, it HAS to be tested on a pure 32-bit processor for me to be able to put the supported stamp on it.

Edit To clarify what I mean about the mmcblk. When CREATING a backup on testing branch version.: The partition table gets created when dd:in the boot partition (it runs dd until the root partition starts, plus a few megabytes, this is why I think it would "just work" if the "boot disks" were to be located BEFORE root)

Then the root partition gets removed. If GPT: sgdisk "$LOOP" -d "$LOCAL_ROOT_PARTN" 2>&1 (row 1238) If MBR: sfdisk --delete -f "$LOOP" "$LOCAL_ROOT_PARTN" 2>&1 (row 1254)

It then recreates the partition by expanding it to cover 100% of the partition on row 1278: parted -s -a none "$LOOP" unit B mkpart primary "$LOCAL_ROOT_START" 100% 2>&1

The last row is what we would have to work with, finding out the the "end bit" of the partition and set that instead of 100% is not much of a problem, the problem is what to do with the "boot disks" AFTER the partition is made, we would have to somehow dd with an offset after the root partition, and I have zero clue if that is even possible.

Please feel free to brainstorm here...

[framps]

Feedback is what helps me advance the script. :)

Fully agree. Valuable feedback helps to enhance a tool and fix any issues. My backup tool got most of it's features by feedback and feature requests. Crossing fingers this also will happen for your tool.

I am sorry about that!

NP. Glad we cleared our misunderstandings :smiley:

I have spent some time thinkgin about the cm4,

Same for me. Maybe it was wrong to make you aware of the CM4. As you wrote it requires some bit of trickery. I unfortunately though it maybe a low hanging fruit. I don't need this support. I thought this will increase the user community of shrink-backup. So it may be a better strategy to wait for somebody to ask for CM4 support with NVMe and eMMC anmd don't spend any time until CM4 support is asked for.

If you're still keen to enable CM4 support I'm willing to execute some basic tests on my CM4 for you. That way you can state in your doc CM4 with eMMC and NVMe works - but it's not fully tested. Not sure it's worth to spend any time on this - but it's your decision :wink:

For nvme, the backup should go through, but how it will work writing that img to an sd-card and boot, I have no idea

In general there is no difference whether you run your system on /dev/sdx, /dev/mcblkx or /dev/nvmexpy. You only have to make sure these devices are discovered correctly. Then theses backups can be restored on any HW if the HW supports the OS (you will not be able to boot a 64bit OS on a 32 bit RPi2).

The problem with backing up the mmcblk is that the "boot disks" are AFTER the root partition, bot before. If they were either before or directly after the boot partition, I actually think the script would work. But again, I have no idea how the boot process works on a cm4. :(

As you know I have my own backup tool and there you can backup of CM4 eMMC and restore it on a SD card and the system boots successfully. You also can restore the backup on the eMMC again and everything works fine even these two additional boot partitions are not saved and restored. It looks like these partitions are created by some Raspberry firmware of the kernel when the CM4 boots from eMMC.

I CAN NOT TEST these things, I would have to rely on you 100%, and it feels bad from my side, not because I don't trust you, but because I put a lot of pressure on you...

As I wrote before: I'm willing to execute some basic tests to test whether shrink-backup in general works on a CM4. But I'm not willing to execute a full test. I have still some open feature requests on my backup tool I have to work on :wink:

If you want to open the other issue again, I will, but tbh, I think focusing on this would be way more interesting and useful.

I agree. I have a spare RPi2 I can use to execute tests with a 32 bit OS on a 32 bit HW. But you first should execute 32 Bit OS tests on your RPi3+ and when you're done I just execute some final tests on the real 32 bit HW on my side to verify everything works on 32 bit HW. I frankly don't expect any issues there.

To clarify what I mean about the mmcblk.

I'm not sure I get you here. You're talking about how to expand the root partition and I understand we're talking about the 32 bit OS :confused:

Please feel free to brainstorm here...

NP. But you should make sure we're both on the same page :wink:

[UnconnectedBedna]

Same for me. Maybe it was wrong to make you aware of the CM4. As you wrote it requires some bit of trickery. I unfortunately though it maybe a low hanging fruit. I don't need this support. I thought this will increase the user community of shrink-backup. So it may be a better strategy to wait for somebody to ask for CM4 support with NVMe and eMMC anmd don't spend any time until CM4 support is asked for.

Well, the flood doors are open now, impossible to tell my brain to drop this now. xD

If you're still keen to enable CM4 support I'm willing to execute some basic tests on my CM4 for you. That way you can state in your doc CM4 with eMMC and NVMe works - but it's not fully tested. Not sure it's worth to spend any time on this - but it's your decision 😉

No worries. I have already started looking into getting a cm4, but it is pretty far down the prio list. But I will ofc keep you in mind! :) Top of the list is a rpi5 with a sata hat so I can have my urbackup running directly to the spinning disks via sata interface rather than connected via usb. :) https://geekworm.com/products/x1009

As you know I have my own backup tool and there you can backup of CM4 eMMC and restore it on a SD card and the system boots successfully. You also can restore the backup on the eMMC again and everything works fine even these two additional boot partitions are not saved and restored. It looks like these partitions are created by some Raspberry firmware of the kernel when the CM4 boots from eMMC.

Oh! So your backup works, but it ignores those partitions (disks) and only creates boot and root from the mmcblk into an img file? I am talking about when you make backups FROM mmcblk, the nvme should work just fine with shrink-backup, it is ONLY the mmcblk that should fail.

I should take a look at how you do things on your repo.

I dropped all things involving your rpi3 here in this thread, just so we don't talk around eachother here, I am ONLY talking about cm4 and the 2 different devices you can backup from, mmcblk and nvme.

Thank you again for giving me your time, I highly appreciate it!

Edit Hope you see this.. xD

I have been looking at your script, but not as a competitor, but as something this script is NOT aiming to become. Ie yours is way more user friendly than what shrink-backup is or most likely ever will be. The main goal of this project is to keep it as super minimal as possible but still work on most linux SBC distros. Witch is a good thing, then we don't have to get pissy at eachother thinking one is better than the other, because we aim at different users. You know, ego and all that childish bs.. :slightly_smiling_face:

But I have gotten a request from a distribution (I wont mention name in case I screw this up, the mentioned health issues f.ex) if I would like to create a backup that should be shipped with the distro as the standard backup tool, and they want a graphical frontend just like you, so maybe that version will be a competitor for you. xD It should already have been in the works, but my health has sadly put up roadblocks for me.

But good job on your backup utility! It looks really professional!

Edit 2 https://www.linux-tips-and-tricks.de/en/faq#a6

If you use a dd backup this will work for a lrger SD card. But you have to use a Linux tool to expand the second partition if you want to use the whole space

I might have an idea for you there. Assuming you only have boot and root (I see you include other functionalities, but you might be able to use this anyway).

https://github.com/UnconnectedBedna/shrink-backup/blob/main/shrink-backup#L1775 It's the same functionality I mentioned that fails on mmcblk earlier in this thread. This is ran at boot and it simply removes the root partition, then recreates it again filling 100% of the available space.

This obv only works if root is the LAST partition on the device and I did not deep dive how you handle other partitions (but did see you have a method for it), but if they are saved on another file and the "base" img is only boot and root, this should work fine.

Or even, like I mentioned earlier, it's the same thing with this as with the mmcblk. If instead of filling it 100%, define an end bit where the partition ends, then make sure to leave space for the other partitions, create them and then copy the remaining files if you want to restore them TO the same device as root. Again, I do not know how you deal with partition outside of boot and root when they are included in the backup.

I probably misunderstand how this works, and this is just an idea. :)

I have TONS of experience when it comes to resizing, and this is part of it so feel free to use me for knowledge if you want. I'd love to help out where I can.

[framps]

Well, the flood doors are open now, impossible to tell my brain to drop this now. xD

I'm sorry I opened the door :-D

Top of the list is a rpi5 with a sata

:+1: It's a nice HW. But I don't have any need for an RPi5. There is nothing different from an OS point of view and I can use my RPi4 for development an test of raspiBackup.

Oh! So your backup works, but it ignores those partitions (disks) and only creates boot and root from the mmcblk into an img file?

Yes. The partition layout is saved during backup with sfdisk and restored also with sfdisk. There is no information about the two bootpartitions in the sfdisk backup. That's why I'm pretty sure shrink-backup also will work in this case if these two partitions are just ignored by shrink-backup during the backup process.

I should take a look at how you do things on your repo.

Feel free to check the code. But raspiBackup works differently that shrink-backup. Tools which work with a similar approach like shrink-backup are for example rpi-clone and image backup.

I dropped all things involving your rpi3 here in this thread, just so we don't talk around eachother here, I am ONLY talking about cm4 and the 2 different devices you can backup from, mmcblk and nvme.

:+1:

have been looking at your script, but not as a competitor,

I evaluated your backup script because I just want to know which backup tools for a Raspberry are available and which ones I can suggest to folks who ask for a backup tool. You script works perfectly and different than raspiBackup and has other target audience. It's just the use case you have which requires somebody to use this or that tool for backup. So there is no competition for me. And if there are some synergy effects and one tool inspires another tool and vice versa - that's great and I love this :-)

The main goal of this project is to keep it as super minimal as possible but still work on most linux SBC distros.

In 10 years I got a lot of feedback and requirements which caused raspiBackup to grow and become very flexible but also quite complex :-(. Therefore I created the installer which helps to create a basic configuration very quickly for folks which are not very familiar with Linux. (You know which OS they are instead familiar with ;-) )

and they want a graphical frontend just like you, so maybe that version will be a competitor for you. xD

No worries. It's important to use some backup tool but doesn't matter which one. It just has to match with the individual use case.

But good job on your backup utility! It looks really professional!

Thank you very much :-)

I might have an idea for you there.

Thank you very much for the link. I know backup tools autoexpand the last partition during boot or in a different way. I frankly don't suggest to use dd and therefore didn't spent any time on the autoexpansion for an image created by dd. I suggest to use either tar or rsync (which is the default) and there the second partition (in next release the last partition so you can have more than two partitions) is autoexpanded. But I will check your code. Maybe I get some idea how to improve my autoexpansion ;-)

Again, I do not know how you deal with partition outside of boot and root when they are included in the backup.

Initially raspiBackup only supported 2 partitions (normal mode). Later on the so called partition oriented mode was added to support NOOBs images. NOOBs is gone but there are folks out there who used this mode to save either all or a specifiied partitions with a normal RaspbianOS. You even can backup the OS with it's two partitions only and in a separate backup run the additional data partitions only. Very useful if you want to migrate to a new RaspbianOS release. In next release the last partition will be autoexpanded as in normal mode.

If you are interested in the details let me know.

I have TONS of experience when it comes to resizing, and this is part of it so feel free to use me for knowledge if you want. I'd love to help out where I can.

Great. Thank you very much. As I already wrote it's not on my todo list but maybe I get a feature request which requires some redesign of the autoexpansion and then I'll get in touch with you :-)

[UnconnectedBedna]

This has been bugging me for a while, and I might have an idea.

Could you provide the output of sudo fdisk -l /dev/mmcblk0

I want to see exactly where those "boot disks" are located.

[framps]

Sure :smiley:

Here we are:

pi@raspberrypi-bookworm-64-lite-cm4-emmc:~ $ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0  29.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0  28.6G  0 part /
mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 
nvme0n1      259:0    0 119.2G  0 disk 
├─nvme0n1p1  259:1    0   512M  0 part 
└─nvme0n1p2  259:2    0 118.7G  0 part 
pi@raspberrypi-bookworm-64-lite-cm4-emmc:~ $ sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 29.12 GiB, 31268536320 bytes, 61071360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xee4b8711

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192  1056767  1048576  512M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      1056768 61063167 60006400 28.6G 83 Linux

[UnconnectedBedna]

Thanks.

And this? sudo parted /dev/mmcblk0 print

[framps]

pi@raspberrypi-bookworm-64-lite-cm4-emmc:~ $ sudo parted /dev/mmcblk0 print
Model: MMC BJTD4R (sd/mmc)
Disk /dev/mmcblk0: 31.3GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   31.3GB  30.7GB  primary  ext4

[UnconnectedBedna]

Ok, this makes little sense to me now.

I really need a new log file to work with here. It's impossible without one.

Please run the script while booted with mmcblk0 (with the latest changes on the testing branch) and provide the log file of the failing backup. And please use the git version so we don't have to involve the untested changes by installing with curl.

Because before it was failing with that parted line, but it seems to work just fine now so a re-evaluation of what is going on is needed. :)

[UnconnectedBedna]

You dislike when I edit, so lets make a new post..

When it inevitably fails, try changing this line with this dirty trick..

line 2062

- LOCAL_DEV_PATH=$(lsblk -no ptuuid,type,path | grep "$LOCAL_DEV_PTUUID" | grep 'disk' | awk '{print $3}')
+ LOCAL_DEV_PATH=$(lsblk -no ptuuid,type,path | grep "$LOCAL_DEV_PTUUID" | grep 'disk' | awk '{print $3}' | head -1)

And retry. Maybe I am making things way more complicated than needed...

It is indeed the parted line that fails, but it is failing because this variable catches all 3 boot things while only the first is needed...

[framps]

You dislike when I edit, so lets make a new post..

If you wouldn't have created a new post I wouldn't have get notified of your update :wink:

Looks much better now. The backup succeeded :+1:

shrink-backup.log

[UnconnectedBedna]

FYI, I made a header for using curl on the wiki when installing in case you would want to create the info. I have not made it to testing the curl functionality yet, maybe you have?

If you can confirm a restore works when using -a you can then finally close this issue as solved. :slightly_smiling_face:

[framps]

If you can confirm a restore works when using -a

I don't want do screw up my current OS on the CM4 ...

It's easy to restore to another SD card and test whether the system boots. But it's the internal eMMC I cannot replace ...

I already have raspiBackup in place and a current backup. So I can boot the CM4 from NVMe and will restore the backup on the eMMC. Frankly I don't expect any issue ... but I agree a restore test should confirm your fix.

[UnconnectedBedna]

I don't want do screw up my current OS on the CM4 ...

Ah, yeah, forgot, you didn't have another sd-card..

I actually was mostly interested if the backup succeeded with using -a when running instead of manually adding space.. xD

Keep this open until you have confirmed with a restore to another sd-card and booted. :slightly_smiling_face:

[framps]

I actually was mostly interested if the backup succeeded with using -a when running instead of manually adding space.. xD

Ok. I will use -a now :wink: and test again.

[framps]

Booted CM4 on eMMC and then executed:

sudo ./shrink-backup -a /mnt/home/pi/shrink-backup.img # create backup on NVMe partition"

Switched to NVMe with switchOS.sh and executed dd if=~/shrink-backup.img of=/dev/mmcblk0 bs=2MiB

Switched back with switchOS.sh to eMMC and booted the restored eMMC OS successfully.