shrink-backup with 'f2fs'?

UnconnectedBedna / shrink-backup

A utility to backup SBC:s (like Raspberry pi) into minimal bootable img files

Other

41 stars 4 forks source link

shrink-backup with 'f2fs'? #31

Closed sneumeister closed 3 months ago

sneumeister commented 5 months ago

Hello,

My RaspberryPi runs since years under "f2fs" (Flash-Friendly File System). It seems your scripts relys on btrfs?

Is there a change to work with f2fs? I see so many btrfs command in the scripts doing not know the exact function...

2024-04-17 22:09:43 [DEBUG] - Running: mkfs.btrfs -m single -L  -f -v /dev/loop0p2
./shrink-backup: Zeile 1162: mkfs.btrfs: Kommando nicht gefunden.
------------------------------------------------------------------------------
2024-04-17 22:09:43 [INFO]  - Creating temp directory
2024-04-17 22:09:43 [DEBUG] - Running: mktemp -d -t backup-XXX
2024-04-17 22:09:43 [DEBUG] - TMP_DIR=/tmp/backup-rMk
2024-04-17 22:09:43 [INFO]  - Creating btrfs subvolumes
2024-04-17 22:09:43 [DEBUG] - Running: mount -o compress=zstd /dev/loop0p2 /tmp/backup-rMk
------------------------------------------------------------------------------
2024-04-17 22:09:44 [ERROR] - ROOT MOUNT FAILED:
mount: /tmp/backup-rMk: wrong fs type, bad option, bad superblock on /dev/loop0p2, missing codepage or helper program, or other error.
------------------------------------------------------------------------------

UnconnectedBedna commented 5 months ago

The script has experimental support for btrfs but is aimed at ext4. It will not work for f2fs. The reason it goes for btrfs is if it is NOT ext4, it just assumes it's btrfs. xD

I have never looked into f2fs. So if you make a good case on why I should implement it (I have for example no idea if this is used a lot or just 1% of users) and label this issue as "enhancement", I might look into it. :nerd_face:

sneumeister commented 4 months ago

I don't know how many use F2FS. I came across several years ago and made good experiences with it (compared with the many "dying SD card" issues I read in different forums. To be honest I didn't think about filesystem... I was looking for a tool to backup a running system without removing the SD card. When I found your script and wanted to give a try I got error messages. So I came here... ;-)

UnconnectedBedna commented 4 months ago

I looked a tiny bit into f2fs. There are limitations, shrinking an existing img seems to be out of the question for example, you would have to create a new img instead.

Also, how did you install it? Did you use a script? Or did you do something like this? https://home.strw.leidenuniv.nl/~dullaart/programming/raspberrypi/2022/01/12/raspi-f2fs-image.html

I'm not giving any promises here, just lightly looking into how much work it would be for me to implement.

sneumeister commented 4 months ago

Installation is quite easy: burn Debian image to SD-Card... start the system... basic Debian set-up... expand filesystem (ext4) to full SD-card size. Shutdown und remove card. Start Debian am my Desktop PC... insert card... copy root-partition content to some temporary space. mkfs.f2fs ...edit fstab and edit cmdline.txt. Then the SD-card is ready for the raspberrypi. No scripts but hand-made... ;-)

UnconnectedBedna commented 4 months ago

That was the route I did with the script as well, ie it does EXACTLY that (except the whole "copy to another location first", it formats the img file to f2fs and then copies directly to that file) if using the --f2fs tag.

It took about 3 minutes to run on fresh raspebrry pi lite install. Tested it and yes, after restore of the img and boot, it is f2fs. :) Felt like the boot took about double the time with f2fs compared to ext4, but maybe that gets better after first boot, or I am imagining things. But the conversion works. :smile:

I have not fixed autoexpansion, and this is ONLY a conversion method as of now. But I have a feeling autoexpansion is just one more line of code in the autoexpansions script. And the backup funtionality, similar, I have a feeling it will not be too complicated to implement, when I have time...

So for information. Converting a live running system into f2fs now works by using --f2fs. Autoexpansion does not work yet (run script with -e for now, or manually add [extra space] with the script), and you can NOT make a backup of the running f2fs yet, only the conversion is implemented.

All this on testing branch.

Will keep this thread updated of the progress.

UnconnectedBedna commented 4 months ago

Update:

I have now looked into this, and there are some things I can not manage for f2fs.

I will let user have f2fs and make backups just like on ext4, but I can not implement any kind of resizing after creation. It seems, unlike ext4, to be a limitation on the filesystem that will not let it be resized when mounted. And since it is hard to unmount root, I simply can not figure out a way to do this on rpiOS.

If someone can enlighten me how to run something on a rpi before root is mounted, or somehow unmount > resize > remount, I can make it happen. I can put the resize script on the boot partition and run it there, IF systmd (or another way to initialize the script) and bash is available. And that whole situation make everything WAY more complicated.

The partition is no issue, I can resize that, it is ONLY the filesystem.

$ sudo parted /dev/mmcblk0 print

Model: SD  (sd/mmc)
Disk /dev/mmcblk0: 8053MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   8053MB  7512MB  primary  f2fs

$ df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            3.6G     0  3.6G   0% /dev
tmpfs           782M  1.2M  780M   1% /run
/dev/mmcblk0p2  2.9G  2.6G  331M  89% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M   16K  5.0M   1% /run/lock
/dev/mmcblk0p1  510M   64M  447M  13% /boot/firmware
tmpfs           781M     0  781M   0% /run/user/1000

$ sudo fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 7.5 GiB, 8053063680 bytes, 15728640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x98535d44

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192  1056767  1048576  512M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      1056768 15728639 14671872    7G 83 Linux

$ sudo resize.f2fs -t 14671872 /dev/mmcblk0p2
Info: Mounted device!
        Error: Not available on mounted device!

So I think the solution is that the user either expands the filesystem themselves on another system after an img restore, or simply set the img size with [extra space] on creation. But that kinda beats the purpose of smaller img size, but it WILL still be light years faster, since it does not rely on dd, but rsync:s ONLY THE DATA.

But feedback and ideas will be listened to here, if anybody has any ideas. Otherwise the above is what will be implemented.

sneumeister commented 4 months ago

Ok, downloaded code of testing branche and run...........

$ sudo bin/shrink-backup-testing  -let /mnt/netbackup/rpi-backuo_2024-05-06.img 1024
## Debugging requested, writing to log file bin/shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
du: Zugriff auf '/home/pi/smadata/SBFspot.db-wal' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/home/pi/smadata/SBFspot.db-shm' nicht möglich: Datei oder Verzeichnis nicht gefunden

##############################################################################
# A backup will be created at /mnt/netbackup/rpi-backuo_2024-05-06.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 263MiB
# Estemated root usage: 11565MiB
# Manually added space: 1024MiB
# Total img size: 12853MiB with 1024MiB extra space included.
##############################################################################
Do you want to continue? [y/n] y
## Creating bootsector...
273628672 bytes (274 MB, 261 MiB) copied, 119 s, 2,3 MB/s
540927+0 Datensätze ein
540927+0 Datensätze aus
276954624 bytes (277 MB, 264 MiB) copied, 123,433 s, 2,2 MB/s
## Resizing img file...
## Looping img file...
## Removing root partition...
## Recreating root partition...
parted: invalid token: f2fs
Error: Expecting a file system type.
## PARTED FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 08.52

The -wal und -shm may be created during inserting PV-Inverter data into SQLite database. But....

parted: invalid token: f2fs
Error: Expecting a file system type.

Less the log-file.....

##############################################################################
2024-05-06 19:22:22 [INFO]  - Debugging requested, writing to log file bin/shrink-backup.log
2024-05-06 19:22:22 [INFO]  - Zoom speed NOT requested, setting SLEEPING=sleep 1 | ZOOM=false
2024-05-06 19:22:23 [INFO]  - f2fs root filesystem detected
2024-05-06 19:22:23 [DEBUG] - FSTYPE=f2fs
2024-05-06 19:22:24 [DEBUG] - LOCAL_DEV_PTUUID=fda627e9 | LOCAL_DEV_PATH=/dev/mmcblk0
2024-05-06 19:22:25 [INFO]  - f2fs filesystem or conversion to f2fs requested, disabling autoexpansion
------------------------------------------------------------------------------
2024-05-06 19:22:25 [DEBUG] - PARTITION_TABLE=msdos
2024-05-06 19:22:25 [DEBUG] - UPDATE=false
2024-05-06 19:22:25 [DEBUG] - RESIZE2FS_RUN=false
2024-05-06 19:22:25 [DEBUG] - PROMPTS=true
2024-05-06 19:22:26 [DEBUG] - EXCLUDE_FILE=true
2024-05-06 19:22:26 [DEBUG] - AUTOEXPAND=false
2024-05-06 19:22:26 [DEBUG] - RSYNC_DELETE=--delete
2024-05-06 19:22:26 [DEBUG] - F2FS=false
------------------------------------------------------------------------------
2024-05-06 19:22:26 [INFO]  - -f selected by user, using bin/exclude.txt
2024-05-06 19:22:26 [DEBUG] - bin/exclude.txt exists
2024-05-06 19:22:26 [INFO]  - Running function: make_img
------------------------------------------------------------------------------
2024-05-06 19:22:26 [INFO]  - Running function: get_dev_variables
2024-05-06 19:22:26 [INFO]  - Separate boot partition detected
2024-05-06 19:22:27 [DEBUG] - LOCAL_DEV_BOOT_PATH=/dev/mmcblk0p1 | LOCAL_DEV_ROOT_PATH=/dev/mmcblk0p2
2024-05-06 19:22:27 [DEBUG] - LOCAL_BOOT_UUID=F661-303B | LOCAL_ROOT_UUID=dd3ebcd1-6e31-4ecb-b2be-12f76140e401 | LOCAL_ROOT_PARTUUID=fda627e9-02
2024-05-06 19:22:28 [DEBUG] - BOOT_PATH=/boot
2024-05-06 19:22:28 [DEBUG] - LOCAL_ROOT_PARTN=2
2024-05-06 19:22:28 [INFO]  - Calculating size for dd to cover bootsector in blocks (512B block size) and adding 256 blocks to overlap into root (only used in img creation)
2024-05-06 19:22:28 [DEBUG] - Running: fdisk -lo start /dev/mmcblk0 | tail -1
2024-05-06 19:22:29 [DEBUG] - LOCAL_ROOT_START=540672 blocks | LOCAL_BOOTSECTOR=540671 blocks | LOCAL_DDBOOTSECTOR=540927 blocks
2024-05-06 19:22:29 [DEBUG] - LOCAL_ROOT_START=276824064 bytes | LOCAL_BOOTSECTOR=276823552 bytes
2024-05-06 19:22:29 [DEBUG] - BLOCKSIZE= bytes
2024-05-06 19:22:29 [INFO]  - f2fs filesystem detected, using du (+512MiB) instead to calculate recommended root size
2024-05-06 19:26:33 [DEBUG] - LOCAL_RESIZE2FS_MIN=10981406249 bytes
2024-05-06 19:26:33 [INFO]  - Calculating TOTAL (space needed for files on root) by adding LOCAL_USED_SPACE and ADDED_SPACE
2024-05-06 19:26:34 [DEBUG] - LOCAL_USED_SPACE=12127158272 bytes | ADDED_SPACE=1073741824 bytes
2024-05-06 19:26:34 [DEBUG] - TOTAL=13200900096 bytes
2024-05-06 19:26:34 [INFO]  - Calculating .img file size by adding LOCAL_BOOTSECTOR to TOTAL
2024-05-06 19:26:34 [DEBUG] - TRUNCATE_TOTAL=13477723648 bytes
2024-05-06 19:28:54 [INFO]  - Do you want to continue? [y/n]
2024-05-06 19:28:54 [INFO]  - Y or y pressed to confirm
------------------------------------------------------------------------------
2024-05-06 19:28:55 [INFO]  - Using dd to create bootsector
------------------------------------------------------------------------------
2024-05-06 19:30:59 [DEBUG] - Running: dd bs=512 count=540927 if=/dev/mmcblk0 of=/mnt/netbackup/rpi-backuo_2024-05-06.img conv=noerror,sync status=progress
540927+0 Datensätze ein
540927+0 Datensätze aus
276954624 bytes (277 MB, 264 MiB) copied, 123,433 s, 2,2 MB/s
------------------------------------------------------------------------------
2024-05-06 19:31:01 [INFO]  - Using truncate to resize img file to 12853MiB
2024-05-06 19:31:01 [DEBUG] - Running: truncate --size=13477723648 /mnt/netbackup/rpi-backuo_2024-05-06.img
2024-05-06 19:31:02 [DEBUG] - Running function: do_loop
2024-05-06 19:31:02 [DEBUG] - LOOP=/dev/loop0
2024-05-06 19:31:03 [DEBUG] - Running: losetup -P /dev/loop0 /mnt/netbackup/rpi-backuo_2024-05-06.img
2024-05-06 19:31:04 [DEBUG] - IS_LOOPED=true
2024-05-06 19:31:04 [INFO]  - Running function: set_img_variables
2024-05-06 19:31:05 [INFO]  - Separate boot partition detected
2024-05-06 19:31:07 [DEBUG] - IMG_DEV_BOOT_PATH=/dev/loop0p1 | IMG_DEV_ROOT_PATH=/dev/loop0p2
2024-05-06 19:31:08 [INFO]  - Using sfdisk to remove root partition
2024-05-06 19:31:08 [DEBUG] - Running: sfdisk --delete -f /dev/loop0 2
2024-05-06 19:31:11 [INFO]  - Using parted to recreate root partition
2024-05-06 19:31:11 [DEBUG] - Running: parted -s -a none /dev/loop0 unit B mkpart primary f2fs 276824064 100%
------------------------------------------------------------------------------
2024-05-06 19:31:12 [ERROR] - PARTED FAILED:
parted: invalid token: f2fs
Error: Expecting a file system type.
------------------------------------------------------------------------------
2024-05-06 19:31:12 [ERROR] - Cleanup function called with non zero exit code: exit 1
2024-05-06 19:31:12 [DEBUG] - Removing loop in cleanup function: losetup -d /dev/loop0
2024-05-06 19:31:13 [INFO]  - Elapsed time: 08.52
2024-05-06 19:31:13 [DEBUG] - Exiting script
##############################################################################

I have a GNU Parted 3.2 installed (seems standard in Debian buster).... but F2FS is only supported from version 3.4 .... Maybe a reason for upgrading to "bookworm"... ;-)

UnconnectedBedna commented 4 months ago

You identified the error immediately, thank you for that! :heart: Yes, you probably have to use the latest software for it to work..

BUT... I will take a look at this, and see if I can remove the f2fs option with parted and use a general tag or something instead, because that is the partition, not the filesystem being recreated and there might be ways around it. And I REALLY want the script to be back compatible, even if it is "just" for f2fs.

I also removed the functionality with -e completely if f2fs is detected, so you don't actually have to run it with -e. :smile:

UnconnectedBedna commented 4 months ago

There, try now. :smile:

After reading the documentation for parted i got reminded there are 2 different commands, mkpart & mkpartfs, and I use the former one. And with that I actually don't need to (but can) specify the filesystem, and since the fsystem is created by formatting the entire partition, I just removed it.

mkpart part-type [fs-type] start end Make a part-type partition with filesystem fs-type (if specified), beginning at start and ending at end (by default in megabytes). fs-type can be one of "fat16", "fat32", "ext2", "HFS", "linux-swap", "NTFS", "reiserfs", or "ufs". part-type should be one of "primary", "logical", or "extended".

mkpartfs part-type fs-type start end Make a part-type partition with filesystem fs-type beginning at start and ending at end (by default in megabytes). Using this command is discouraged. Instead use mkpart to create an empty partition, and then use external tools like mke2fs(8) to create the filesystem.

It works on my converted f2fs here, but could you confirm on yours, if you still have the old parted version installed?

Also, I think I will change the du to doing calculations with something like stat -f / or just a simple df instead, because the initial du takes forever on mine. ~~I could just increase the safety margin by 128MiB or so and it would probably still be safe.~~

sneumeister commented 4 months ago

Hello, Your fast.... ;-) Ok, another try......

:~ $ sudo bin/shrink-backup-testing  -lt /mnt/netbackup/rpi-backuo_2024-05-07.img 2048
## Debugging requested, writing to log file bin/shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...

##############################################################################
# A backup will be created at /mnt/netbackup/rpi-backuo_2024-05-07.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 263MiB
# Estemated root usage: 11623MiB
# Manually added space: 2048MiB
# Total img size: 13935MiB with 2048MiB extra space included.
##############################################################################
Do you want to continue? [y/n] y
## Creating bootsector...

275944960 bytes (276 MB, 263 MiB) copied, 125 s, 2,2 MB/s
540927+0 Datensätze ein
540927+0 Datensätze aus
276954624 bytes (277 MB, 264 MiB) copied, 128,141 s, 2,2 MB/s
## Resizing img file...
## Looping img file...
## Removing root partition...
## Recreating root partition...
## Formatting filesystem...
mkfs.f2fs: invalid option -- 'U'
        Error: Unknown option ?

Usage: mkfs.f2fs [options] device [sectors]
[options]:
  -a heap-based allocation [default:0]
  -c [device path] up to 7 devices excepts meta device
  -d debug level [default:0]
  -e [cold file ext list] e.g. "mp3,gif,mov"
  -E [hot file ext list] e.g. "db"
  -f force overwrite the exist filesystem
  -i extended node bitmap, node ratio is 20% by default
  -l label
  -m support zoned block device [default:0]
  -o overprovision ratio [default:5]
  -O [feature list] e.g. "encrypt"
  -q quiet mode
  -s # of segments per section [default:1]
  -S sparse mode
  -t 0: nodiscard, 1: discard [default:1]
  -w wanted sector size
  -z # of sections per zone [default:1]
sectors: number of sectors. [default: determined by device size]
## Creating temp directory...
## Mounting img root partition...
mount: /tmp/backup-ACb: mount(2) system call failed: Die Struktur muss bereinigt werden.
## ROOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 02.46

Hmmm..... installed I have: f2fs-tools/oldoldstable,now 1.11.0-1.1 armhf [installiert]

And mkfs seems to be part of util-linux of which I have installed util-linux/oldoldstable,now 2.33.1-0.1+deb10u1 armhf [installiert]

What was intention of commandline option -U "$UUID"?

UnconnectedBedna commented 4 months ago

Old software strikes again, but this time I am afraid there is nothing I can do, unless I also add and catch the created UUID, then changing cmdline.txt. But I am afraid THAT would be a bit too much, since:

$ mkfs.f2fs --help
Usage: mkfs.f2fs [options] device [sectors]
[options]:
  -a heap-based allocation [default:0]
  -c device1[,device2,...] up to 7 additional devices, except meta device
  -d debug level [default:0]
  -e [cold file ext list] e.g. "mp3,gif,mov"
  -E [hot file ext list] e.g. "db"
  -f force overwrite of the existing filesystem
  -g add default options
  -i extended node bitmap, node ratio is 20% by default
  -l label
  -U uuid
  -m support zoned block device [default:0]
  -o overprovision percentage [default:auto]
  -O feature1[,feature2,...] e.g. "encrypt"
  -C [encoding[:flag1,...]] Support casefolding with optional flags
  -q quiet mode
  -r set checkpointing seed (srand()) to 0
  -R root_owner [default: 0:0]
  -s # of segments per section [default:1]
  -S sparse mode
  -t 0: nodiscard, 1: discard [default:1]
  -T timestamps
  -w wanted sector size
  -z # of sections per zone [default:1]
  -V print the version number and exit
sectors: number of sectors [default: determined by device size]

$ mkfs.f2fs -V
mkfs.f2fs 1.15.0 (2022-05-13)

I am ready and very much willing to make the script as backwards compatible as possible, however in this case, it would be way to much. I'm afraid you will have to upgrade to a newer version for it to work. :(

What was intention of commandline option -U "$UUID"?

UUID it the unique identifer linux uses to identify the partition (simply described) so it has to be the same as the one on your root partition, or a restored img will end up in a kernel panic when it cant find the partition because the UUID has changed.

I think you wrote Buster earlier, and IIRC that was the one before Bullseye, and we are now on Bookworm. Your os is VERY old...

sneumeister commented 4 months ago

Hello, Yes, BUSTER is very old... but still working... ;-) With fdisk -l you can also check the UUID and PARTUUID on devices and *.img files.

I found that out, when I took the SD-Card from the Raspi to my PC for backup the card before upgrading debian distribution. My PC card reading seems to be buggy and damaged the card (or at least partition).. so I am struggeling to get card running again since no full backup... :-(

UnconnectedBedna commented 4 months ago

Sorry to hear that. If the card is corrupted, there is nothing to do. A lot if times, you can still read the card, but not write to it before it becomes completely corrupted. If you are still able to access the card manually, you can always try dd, but there is no guarantee it will create a functioning img. Edit You might even want to try gddrescue:

sudo apt update && sudo apt upgrade -y
sudo apt install gddrescue
sudo ddrescue -f -n /dev/mmcblk0 /path/to/backup.img

The /dev/mmcblk is where the sd-card DEVICE is, not any partition. On another linux system where you use an adapter it would most likely be a /dev/sdX. You can do this with the sd-card unmounted if you use the adapter method. end edit

But nothing of that has anything to do with the failing UUID or PARTUUID in the script, that is because your software is outdated on Buster. You can also just pull the latest version of mkfs from the internet and use instead, but since your card seems corrupted, a backup is probably too late for this situation. I do not recommend you relying on that.

As a side note, IMHO you should not upgrade, you install the new release. That is kinda the whole point with fixed point releases. There ARE methods to upgrade, but I strongly discourage doing that. Linux is not windows, there are fixed point releases, and there are rolling releases. RpiOS is a fixed point release.

Better is to make backups of whatever you use the rpi for so you have the configs/settings (for example your /etc & /home), then install the new release and restore the settings as they were on the previous install using the backed up files as reference.

But that is MY opinion, you do you.. :slightly_smiling_face:

sneumeister commented 3 months ago

So, it really was the SD card finally dying. At least after some struggling, dd's and loop mounts I got most files. When identifying correctly I bought this card 2013... running nearly 11 years for 24/7 with every 10min reading from solar inverter and write to sqlite... not bad (SAMSUNG PRO); let's see how long the new SanDisk Extreme PRO will do.

So I set up new BOOKWORM with the SBFSpot, Node-Red, Mosquitto, OpenVPN.... again root file system in f2fs..... ;-)

Running the testing branch of shrink-backup:

~/bin/shrink-backup $ sudo ./shrink-backup -t -a  -a -l /mnt/netbackup/2024-05-20_bookworm.img 1024
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...

##############################################################################
# A backup will be created at /mnt/netbackup/2024-05-20_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: true
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 515MiB
# Estemated root usage: 6313MiB
# Auto calculated size (root partition): 6825MiB
# Total img size: 7341MiB
##############################################################################
Do you want to continue? [y/n] Y
## Creating bootsector...
539070464 bytes (539 MB, 514 MiB) copied, 150 s, 3.6 MB/s
1057023+0 records in
1057023+0 records out
541195776 bytes (541 MB, 516 MiB) copied, 152.636 s, 3.5 MB/s
## Resizing img file...
## Looping img file...
## Removing root partition...
## Recreating root partition...
## Formatting filesystem...

    F2FS-tools: mkfs.f2fs Ver: 1.15.0 (2022-05-13)

Info: Disable heap-based policy
Info: Debug level = 0
Info: Label = system
Info: Trim is enabled
        Error: Failed to get the device stat!
## Creating temp directory...
## Mounting img root partition...
mount: /tmp/backup-ptl: wrong fs type, bad option, bad superblock on , missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
## ROOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 03.08

Recommended dmesg -T shows:

[Mon May 20 22:36:41 2024] loop0: detected capacity change from 0 to 15035079
[Mon May 20 22:36:42 2024]  loop0: p1 p2
[Mon May 20 22:36:42 2024] loop0: p2 size 248680448 extends beyond EOD, truncated
[Mon May 20 22:36:42 2024]  loop0: p1 p2
[Mon May 20 22:36:42 2024] loop0: p2 size 248680448 extends beyond EOD, truncated
[Mon May 20 22:36:46 2024]  loop0: p1
[Mon May 20 22:36:46 2024]  loop0: p1
[Mon May 20 22:36:49 2024]  loop0: p1 p2
[Mon May 20 22:36:56 2024]  loop0: p1 p2
[Mon May 20 22:36:57 2024] : Can't lookup blockdev
[Mon May 20 22:36:57 2024] : Can't lookup blockdev
[Mon May 20 22:36:57 2024] : Can't lookup blockdev
[Mon May 20 22:36:57 2024] fuseblk: Bad value for 'source'

So I installed fuse sudp apt install fuse and re-run shrink-backup...............

Now running.... and showing a lot of files being transferred... Let's see when it finishes.

But I think some "requisition tests" may help to identify obstacles.

sneumeister commented 3 months ago

Since SSH connection terminated yesterday, another try... with error.....

$ sudo ./shrink-backup -atl /mnt/netbackup/2024-05-20_bookworm.img 2048
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
lsblk: /dev/mmcblk0p1
/dev/loop0p1: not a block device

##############################################################################
# A backup will be created at /mnt/netbackup/2024-05-20_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: true
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 515MiB
# Estemated root usage: 6467MiB
# Auto calculated size (root partition): 6979MiB
# Total img size: 7495MiB
##############################################################################
Do you want to continue? [y/n] Y
WARNING!!! WARNING!!! WARNING!!!
/mnt/netbackup/2024-05-20_bookworm.img
FILE ALREADY EXISTS!!!
Do you want to overwrite? [y/n] y
## Removing old img file...
## Creating bootsector...
539152896 bytes (539 MB, 514 MiB) copied, 197 s, 2.7 MB/s
1057023+0 records in
1057023+0 records out
541195776 bytes (541 MB, 516 MiB) copied, 199.762 s, 2.7 MB/s
## Resizing img file...
## Looping img file...
## Removing root partition...
## Recreating root partition...
## Formatting filesystem...

    F2FS-tools: mkfs.f2fs Ver: 1.15.0 (2022-05-13)

Info: Disable heap-based policy
Info: Debug level = 0
Info: Label = system
Info: Trim is enabled
        /dev/loop1p2 appears to contain an existing filesystem (f2fs).
Info: Segments per section = 1
Info: Sections per zone = 1
Info: sector size = 512
Info: total sectors = 14294671 (6979 MB)
Info: zone aligned segment0 blkaddr: 512
Info: format version with
  "Linux version 6.6.28+rpt-rpi-v6 (debian-kernel@lists.debian.org) (gcc-12 (Raspbian 12.2.0-14+rpi1) 12.2.0, GNU ld (GNU Binutils for Raspbian) 2.40) #1 Raspbian 1:6.6.28-1+rpt1 (2024-04-22)"
Info: [/dev/loop1p2] Discarding device
Info: This device doesn't support BLKSECDISCARD
Info: This device doesn't support BLKDISCARD
Info: Overprovision ratio = 2.410%
Info: Overprovision segments = 171 (GC reserved = 90)
Info: format successful
## Creating temp directory...
## Mounting img root partition...
## Mounting img boot partition...
mount: /tmp/backup-YnY/boot/firmware: special device PATH
/dev/loop1
/dev/loop1p1
/dev/loop1p2 does not exist.
       dmesg(1) may have more information after failed mount system call.
## BOOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 04.31

And the dmesg -T output:


[Tue May 21 20:03:35 2024] loop1: detected capacity change from 0 to 15027135
[Tue May 21 20:03:35 2024]  loop1: p1 p2
[Tue May 21 20:03:35 2024]  loop1: p1 p2
[Tue May 21 20:07:49 2024] loop1: detected capacity change from 0 to 15351439
[Tue May 21 20:07:49 2024]  loop1: p1 p2
[Tue May 21 20:07:49 2024] loop1: p2 size 248680448 extends beyond EOD, truncated
[Tue May 21 20:07:53 2024]  loop1: p1
[Tue May 21 20:07:53 2024]  loop1: p1
[Tue May 21 20:07:56 2024]  loop1: p1 p2
[Tue May 21 20:08:32 2024]  loop1: p1 p2
[Tue May 21 20:08:33 2024] F2FS-fs (loop1p2): Found nat_bits in checkpoint
[Tue May 21 20:08:33 2024] F2FS-fs (loop1p2): Mounted with checkpoint version = 596d1a59
[Tue May 21 20:08:35 2024] PATH
                           /dev/loop1
                           /dev/loop1p1
                           /dev/loop1p2: Can't lookup blockdev

UnconnectedBedna commented 3 months ago

The log from that last backup would be really helpful.

Did you already have the img file looped? because if there were no loop, the loop would be loop0 And I have no idea if how it will work if you have it looped twice, remove the file, recreate, etc.

Now running.... and showing a lot of files being transferred...

Indicates there is nothing wrong with the script, if you managed to get it to work once... But I could be wrong.

Also if you use -a there is no use setting extra space, that then gets set automatically. If you want to use [extra space] do NOT include -a. :slightly_smiling_face:

But a dmesg doesn't really tell me much, I need the log file.

sneumeister commented 3 months ago

Logfile... here we are... shrink-backup.log

UnconnectedBedna commented 3 months ago

When this line is run in the script (line 259): LOCAL_DEV_BOOT_PATH=$(lsblk -lpo mountpoint,path | grep 'boot' | awk '{print $2}') it returns (line 87 AND 88 in your log file, should ONLY be one line):

2024-05-21 20:04:09 [DEBUG] - LOCAL_DEV_BOOT_PATH=/dev/mmcblk0p1
/dev/loop0p1 | LOCAL_DEV_ROOT_PATH=/dev/mmcblk0p2

I was asking if it was looped when you ran the backup, and it seems it was, at least it finds a "boot" on the loop "partition". The script has not looped anything at this point, with errors or when script exits it unmounts everything, removes the loops created and removes temp files/directories. So this is something you have looped outside the script. You mentioned doing this in one of your posts, that's why I suspected it, and your log file seems to confirm that further.

I have been contemplating using something other than lsblk | grep 'boot' for this operation, but I am not sure even that would have solved this for you in this situation, because the file is still looped (and mounted I assume) at ANOTHER point, so...

Remove the loop (or reboot your system) and rerun the script and it should work.

You might also want to pull the latest version from the testing branch, I made a few changes...

sneumeister commented 3 months ago

Re-Running after several reboots (I was struggling a bit with OpenVPN and the new NFTables after setting u p Bookworn). This time there were no errors. But I did not mount *.img to loop and check content nor burnt to SDcard for testing... ;-)

Here is latest Log-File..... shrink-backup.log

UnconnectedBedna commented 3 months ago

I scrolled your log and I do not think that img will boot for the simple reason that your boot partition was not mounted/found when mounting img partitions.

This probably means the files in /boot/firmware was removed on your img. They got transfered in the dd, but when the rsync is backing up, it will REMOVE the files because on your system, where the boot mount should be is just an empty directory. So when the rsync is run, the "syncing" of the boot partition is to remove the files already there.

If you DID have your boot mounted there is something with the script, but I find it unlikely because the only thing it does is to run: if [ $(lsblk | grep -c 'boot') -ne 0 ]; then Ie, it checks for a ANY path containing 'boot' among what you have mounted, in your case the boot is at /boot/firmware (as seen in earlier in the log when checking your fstab). It finds your mount in fstab, but does not seem to be mounted when you ran the backup.

I will try to add a check when mounting, if the boot path variable is not empty but can not find that mount, the script SHOULD fail. I just never thought of that happening. xD It is actually good you point this situation out for me, because this is a dangerous situation, you found a "bug" sir... The user THINKS the backup is valid, the script gives no errors, but I feel pretty confident your entire boot device is excluded so this backup is NOT valid!

If you can rerun the script (new backup, not an update on this), but this time with the boot mounted and see if it works. Then if you can also write that to another sd-card and try to boot that in your rpi, I can close this issue and it will be included in the next update to stable.

Please also provide the output of lsblk in your next post so I can see exactly what you have mounted when running the backup.

I was kinda hoping you would use the conversion method in the script when converting from ext4 to f2fs, but I have run that multiple times and it seems to work every time, so no biggie. :slightly_smiling_face:

But please confirm the script works (including a restore) so I can close this thread. :heart:

Edit I updated testing branch adding a check if the boot partition is found in fstab but not found mounted:

  if [ -n "$BOOT_PATH" ]; then
    if [ $(lsblk | grep -c 'boot') -eq 0 ]; then
      echo '!! BOOT PARTITION NOT MOUNTED ON SYSTEM, PLEASE MOUNT BOOT PARTITION AND RE-RUN THE SCRIPT!!!'
      debug 'ERROR' 'BOOT PARTITION NOT MOUNTED ON SYSTEM, PLEASE MOUNT BOOT PARTITION AND RE-RUN THE SCRIPT!!!'
      exit 1
    fi

sneumeister commented 3 months ago

Hello, another trial under "controlled conditions" (for whatever reason the /boot/firmware was again un-mount and last trial ended with remaining loop-devices.... So this times pre-conditions:

$ losetup -D
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part
└─mmcblk0p2 179:2    0 118.6G  0 part /

==> only SD-card, no loop-devices attached.

$ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=655d03f1-01  /boot/firmware  vfat    defaults          0       2
PARTUUID=655d03f1-02  /               f2fs    defaults,noatime  0       1
diskstation2:/volume1/NetBackup /mnt/netbackup nfs defaults,noauto 0 0

# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

==>according fstab /boot/firmware should be mounted (this was only /boot under BUSTER).

$ ls /boot/firmware
bcm2708-rpi-b.dtb
bcm2708-rpi-b-plus.dtb
bcm2708-rpi-b-rev1.dtb
bcm2708-rpi-cm.dtb
bcm2708-rpi-zero.dtb
bcm2708-rpi-zero-w.dtb
bcm2709-rpi-2-b.dtb
[...]

==> is really mounted.

Furthermore I moved shrink-backup from ~/bin/shrink-backup to folder of backup-target /mnt/netbackup/shrink-backup ; this is the shrink-backup.log no longer in folder to be backed up (whatever may happen you copy/rsync some changing file). Starting shrink-backup from folder /mnt/netbackup/shrink-backup /mnt/netbackup/shrink-backup $ sudo ./shrink-backup -atl ./2024-05-30_shrink-backup_bookworm.img

Logfile after finish..... shrink-backup.log Last lines of shell-output:

Number of files: 195,257 (reg: 167,454, dir: 20,497, link: 7,305, special: 1)
Number of created files: 194,874 (reg: 167,076, dir: 20,492, link: 7,305, special: 1)
Number of deleted files: 0
Number of regular files transferred: 167,020
Total file size: 6.68G bytes
Total transferred file size: 5.35G bytes
Literal data: 5.35G bytes
Matched data: 0 bytes
File list size: 5.76M
File list generation time: 0.036 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.36G
Total bytes received: 3.33M

sent 5.36G bytes  received 3.33M bytes  1.39M bytes/sec
total size is 6.68G  speedup is 1.25
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: true
## ./2024-05-30_shrink-backup_bookworm.img is 8979MiB with a root partition of 8463MiB.
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 09.03

lsblk shows no remaining loops ==> OK. /boot/firmware still mounted ==> OK.

$ fdisk -l 2024-05-30_shrink-backup_bookworm.img
Disk 2024-05-30_shrink-backup_bookworm.img: 8.77 GiB, 9415372288 bytes, 18389399 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x655d03f1

Device                                 Boot   Start      End  Sectors  Size Id Type
2024-05-30_shrink-backup_bookworm.img1         8192  1056767  1048576  512M  c W95 FAT32 (LBA)
2024-05-30_shrink-backup_bookworm.img2      1056768 18389398 17332631  8.3G 83 Linux

$ sudo mount -o loop,offset=4194304 2024-05-30_shrink-backup_bookworm.img /mnt/shr1
$  ls /mnt/shr1
 bcm2708-rpi-b.dtb        bcm2710-rpi-2-b.dtb        bcm2711-rpi-cm4.dtb         cmdline.txt    fixup_db.dat   kernel7l.img       start4x.elf
 bcm2708-rpi-b-plus.dtb   bcm2710-rpi-3-b.dtb        bcm2711-rpi-cm4-io.dtb      config.txt     fixup_x.dat    kernel8.img        start_cd.elf
 bcm2708-rpi-b-rev1.dtb   bcm2710-rpi-3-b-plus.dtb   bcm2711-rpi-cm4s.dtb        fixup4cd.dat   initramfs      kernel.img         start_db.elf
 bcm2708-rpi-cm.dtb       bcm2710-rpi-cm3.dtb        bcm2712d0-rpi-5-b.dtb       fixup4.dat     initramfs7     LICENCE.broadcom   start.elf
 bcm2708-rpi-zero.dtb     bcm2710-rpi-zero-2.dtb     bcm2712-rpi-5-b.dtb         fixup4db.dat   initramfs7l    overlays           start_x.elf
 bcm2708-rpi-zero-w.dtb   bcm2710-rpi-zero-2-w.dtb   bcm2712-rpi-cm5-cm4io.dtb   fixup4x.dat    initramfs8     start4cd.elf      'System Volume Information'
 bcm2709-rpi-2-b.dtb      bcm2711-rpi-400.dtb        bcm2712-rpi-cm5-cm5io.dtb   fixup_cd.dat   issue.txt      start4db.elf
 bcm2709-rpi-cm2.dtb      bcm2711-rpi-4-b.dtb        bootcode.bin                fixup.dat      kernel7.img    start4.elf

==> Seems all there....

$ sudo mount -t f2fs -r  -o loop,offset=541065216 2024-05-30_shrink-backup_bookworm.img /mnt/shr2
$  ls /mnt/shr2/
bin  boot  boot.bak  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  wget.txt

$ ls /mnt/shr2/etc/
adduser.conf            default            idmapd.conf         logcheck        opensc                            rc6.d             sudo.conf
alternatives            deluser.conf       ifplugd             login.defs      openvpn                           rc.local          sudoers
[...]

==> Seems all there....

Writing to SD and boot I can currently not test since not at home. All the backup tests done remotely via SSH.... ;-)

sneumeister commented 3 months ago

Ok, and now "Update" test.........

$  sudo ./shrink-backup -Ul 2024-05-30_shrink-backup_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
## Looping img file...
##############################################################################
# Updating 2024-05-30_shrink-backup_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# Resize operations not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: false
# Bootsector size: 515MiB
# Estemated root usage: 7903MiB
# Total img size: 8979MiB
##############################################################################
Do you want to continue? [y/n] y
## Creating temp directory...
## Mounting img root partition...
mount: /tmp/backup-XCK: wrong fs type, bad option, bad superblock on , missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
## ROOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 00.29

And logfile......... shrink-backup.log

UnconnectedBedna commented 3 months ago

Thank you for taking the time.

To me it looks like your boot is NOT mounted because:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part      <== YOUR BOOT IS NOT MOUNTED
└─mmcblk0p2 179:2    0 118.6G  0 part /

It should say /boot/firmware under mountpoints for the first partition. THIS is how it should look:

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 14.8G  0 disk 
├─mmcblk0p1 179:1    0  512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 14.3G  0 part /

Only your root is mounted, so:

$ ls /boot/firmware
bcm2708-rpi-b.dtb
bcm2708-rpi-b-plus.dtb
bcm2708-rpi-b-rev1.dtb
bcm2708-rpi-cm.dtb
bcm2708-rpi-zero.dtb
bcm2708-rpi-zero-w.dtb
bcm2709-rpi-2-b.dtb

Only means that those files exist in /boot/firmware on your ROOT PARTITION. With boot unmounted, that directory should actually be empty. So that probably means you have somehow reinstalled your boot/kernel without having your boot partition mounted... (maybe in the kernel update from 6.1 > 6.6 I saw happened a few days ago on bookworm) Either way, I have kernel8 on my rpi so...

ls -l /boot/firmware/
total 63304
-rwxr-xr-x 1 root root     1594 May  9 21:02 LICENCE.broadcom
-rwxr-xr-x 1 root root    31760 Apr 22 15:49 bcm2710-rpi-2-b.dtb
-rwxr-xr-x 1 root root    34587 Apr 22 15:49 bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root    33952 Apr 22 15:49 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root    31663 Apr 22 15:49 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root    32929 Apr 22 15:49 bcm2710-rpi-zero-2-w.dtb
-rwxr-xr-x 1 root root    32929 Apr 22 15:49 bcm2710-rpi-zero-2.dtb
-rwxr-xr-x 1 root root    55365 Apr 22 15:49 bcm2711-rpi-4-b.dtb
-rwxr-xr-x 1 root root    55369 Apr 22 15:49 bcm2711-rpi-400.dtb
-rwxr-xr-x 1 root root    38349 Apr 22 15:49 bcm2711-rpi-cm4-io.dtb
-rwxr-xr-x 1 root root    56005 Apr 22 15:49 bcm2711-rpi-cm4.dtb
-rwxr-xr-x 1 root root    52776 Apr 22 15:49 bcm2711-rpi-cm4s.dtb
-rwxr-xr-x 1 root root    80190 Apr 22 15:49 bcm2712-rpi-5-b.dtb
-rwxr-xr-x 1 root root    80631 Apr 22 15:49 bcm2712-rpi-cm5-cm4io.dtb
-rwxr-xr-x 1 root root    80623 Apr 22 15:49 bcm2712-rpi-cm5-cm5io.dtb
-rwxr-xr-x 1 root root    80174 Apr 22 15:49 bcm2712d0-rpi-5-b.dtb
-rwxr-xr-x 1 root root    52476 May  9 21:02 bootcode.bin
-rwxr-xr-x 1 root root      118 Jan 23 21:43 cmdline.txt
-rwxr-xr-x 1 root root     1258 Jan 23 05:49 config.txt
-rwxr-xr-x 1 root root     7303 May  9 21:02 fixup.dat
-rwxr-xr-x 1 root root     5436 May  9 21:02 fixup4.dat
-rwxr-xr-x 1 root root     3204 May  9 21:02 fixup4cd.dat
-rwxr-xr-x 1 root root     8425 May  9 21:02 fixup4db.dat
-rwxr-xr-x 1 root root     8425 May  9 21:02 fixup4x.dat
-rwxr-xr-x 1 root root     3204 May  9 21:02 fixup_cd.dat
-rwxr-xr-x 1 root root    10266 May  9 21:02 fixup_db.dat
-rwxr-xr-x 1 root root    10268 May  9 21:02 fixup_x.dat
-rwxr-xr-x 1 root root 11568635 May 29 18:03 initramfs8
-rwxr-xr-x 1 root root 11570040 May 29 18:03 initramfs_2712
-rwxr-xr-x 1 root root      145 Dec 11 05:54 issue.txt
-rwxr-xr-x 1 root root  9272107 May  9 21:03 kernel8.img
-rwxr-xr-x 1 root root  9274327 May  9 21:03 kernel_2712.img
drwxr-xr-x 2 root root    28672 May  9 21:02 overlays
-rwxr-xr-x 1 root root  2981056 May  9 21:02 start.elf
-rwxr-xr-x 1 root root  2256800 May  9 21:02 start4.elf
-rwxr-xr-x 1 root root   809468 May  9 21:02 start4cd.elf
-rwxr-xr-x 1 root root  3754024 May  9 21:02 start4db.elf
-rwxr-xr-x 1 root root  3004488 May  9 21:02 start4x.elf
-rwxr-xr-x 1 root root   809468 May  9 21:02 start_cd.elf
-rwxr-xr-x 1 root root  4825896 May  9 21:02 start_db.elf
-rwxr-xr-x 1 root root  3728104 May  9 21:02 start_x.elf

Moving on... All I can say is, if you have both root and boot mounted, the backup will most likely work without problems.

The only way to find out is to restore the img and test, but to me it looks like you have your system strangely setup or your boot should NOT randomly become unmounted.

Also, if you pull the latest version from testing, you will now get an error if your boot is not mounted but present in fstab.

So I will only focus on the first backup, the second does not matter since the first was made in error.

Reboot.
DO NOT DO ANYTHING ON YOUR SYSTEM, NO LOOPING, NO MOUNTING, NOTHING...
Run script
Write image to another sd-card
insert sd-card in rpi and test

This is what you have to focus on first and when that works, we can move on and see if there is something wrong with updating the img.

Edit I also recommend you read the documentation on this github page, you can use the script to loop the file, no need for any offsets when mounting (you don't need that anyway, use -p to probe and loop with losetup and they will show up as partitions in lsblk so you can mount the loops directly).

sudo shrink-backup --loop /path/to/backup.img

Then mount with for example: sudo mount /dev/loop0p2 /mnt/shr2 And mount boot partition inside the root with: sudo mount /dev/loop0p1 /mnt/shr2/boot/firmware

sneumeister commented 3 months ago

So downloaded latest test version.

OK, starting with reboot......... Below the SSH log directly after logging in after reboot:

login as: ###
Authenticating with public key ###
Linux george 6.6.28+rpt-rpi-v6 #1 Raspbian 1:6.6.28-1+rpt1 (2024-04-22) armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 30 17:36:31 2024 from ###.###.###.###
###@george:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 118.6G  0 part /
###@george:~ $ sudo mount /mnt/netbackup/
Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /lib/systemd/system/rpc-statd.service.
###@george:~ $ cd /mnt/
###@george:/mnt $ cd netbackup/
###@george:/mnt/netbackup $ cd shrink-backup/
###@george:/mnt/netbackup/shrink-backup $ ls -la
total 6658212
drwxr-xr-x 1 1024 users        122 May 30 17:48 .
drwxrwxrwx 1 root root         122 May 30 13:23 ..
-rw-r--r-- 1 1024 users 9415372288 May 30 16:21 2024-05-30_shrink-backup_bookworm.img
-rw-r--r-- 1 1024 users         74 May 20 22:27 exclude.txt
-rwxr-xr-x 1 1024 users      89349 May 30 17:30 shrink-backup
###@george:/mnt/netbackup/shrink-backup $ sudo ./shrink-backup -atl 2024-05-30_17-56_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...

##############################################################################
# A backup will be created at 2024-05-30_17-56_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: true
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 515MiB
# Estemated root usage: 7938MiB
# Auto calculated size (root partition): 8450MiB
# Total img size: 8966MiB
##############################################################################
## Do you want to continue? [y/n] y

## Creating bootsector...
540239872 bytes (540 MB, 515 MiB) copied, 198 s, 2.7 MB/s
1057023+0 records in
1057023+0 records out
541195776 bytes (541 MB, 516 MiB) copied, 199.578 s, 2.7 MB/s
## Resizing img file...
## Looping img file...
## Removing root partition...
## Recreating root partition...
## Formatting filesystem...

    F2FS-tools: mkfs.f2fs Ver: 1.15.0 (2022-05-13)

Info: Disable heap-based policy
Info: Debug level = 0
Info: Label = system
Info: Trim is enabled
        Error: Failed to get the device stat!
## Creating temp directory...
## Mounting img root partition...
mount: /tmp/backup-cE7: wrong fs type, bad option, bad superblock on , missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
!! ROOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 04.00

And shrink-backup.log

UnconnectedBedna commented 3 months ago

2024-05-30 18:01:11 [DEBUG] - IMG_DEV_BOOT_PATH= | IMG_DEV_ROOT_PATH=

Yeah, you get that sh*t too, it is something that happens random, there is another bug report about it happening on dietpi, but the problem is I can not reproduce it on my systems any more...

Ok, we are going to do this in a special way.

shrink-backup-f2fs.txt

Download that version, I had to name it .txt or github would not let me upload, change if you want.. xD

Make it executable chmod +x shrink-backup-f2fs

Then run the script again. When it says: Press [Enter] key to continue... Open another terminal and type:

lsblk -o path,uuid /dev/loop0 | grep '3F92-1DB9' | awk '{print $1}'
lsblk -o path,uuid /dev/loop0 | grep '66378ef3-17b3-4f70-b069-2fa3e5b1f9c4' | awk '{print $1}'

Those are the lines that fail in the script, so lets see what you get in a terminal when script is at work.

Thanks again for taking the time. :slightly_smiling_face:

sneumeister commented 3 months ago

Hello, I downloaded the "shrink-backup-f2fs.txt" removed the *.txt and made executable. Then run the script with NO reboot before.

## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
lsblk: : not a block device

The /boot/firmware is un-mounted.... what I meant in earylier post... "suddenly" unmounted.

 $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part
└─mmcblk0p2 179:2    0 118.6G  0 part /

What I did before: After last run of your script (the last post of me), I did not check whether '/boot/firmware' was still mounted. I just disconnected from SSH and reconnected after your last post. Could there be a connection between fail-exit of script and mount status?

OK:

$ sudo mount /boot/firmware
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 118.6G  0 part /

Re-run the script:

$ sudo ./shrink-backup-f2fs -alt 2024-05-30_18-47_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...

##############################################################################
# A backup will be created at 2024-05-30_18-47_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: true
# Autoexpand filesystem at boot: false
# Use exclude.txt: true
# Bootsector size: 515MiB
# Estemated root usage: 7885MiB
# Auto calculated size (root partition): 8397MiB
# Total img size: 8913MiB
##############################################################################
## Do you want to continue? [y/n] y

## Creating bootsector...
539525120 bytes (540 MB, 515 MiB) copied, 169 s, 3.2 MB/s
1057023+0 records in
1057023+0 records out
541195776 bytes (541 MB, 516 MiB) copied, 173.387 s, 3.1 MB/s
## Resizing img file...
## Looping img file...
Press [Enter] key to continue...

Output from other terminal:

 $ lsblk -o path,uuid /dev/loop0 | grep '3F92-1DB9'
/dev/loop0p1 3F92-1DB9

 $ lsblk -o path,uuid /dev/loop0 | grep '66378ef3-17b3-4f70-b069-2fa3e5b1f9c4'
/dev/loop0p2 66378ef3-17b3-4f70-b069-2fa3e5b1f9c4

$ lsblk -o path,uuid /dev/loop0
PATH         UUID
/dev/loop0
/dev/loop0p1 3F92-1DB9
/dev/loop0p2 66378ef3-17b3-4f70-b069-2fa3e5b1f9c4

Seem both to exist.... So [Enter] in the first terminal to continue.... And seemed to finish w/o errors....

Number of files: 195,273 (reg: 167,470, dir: 20,497, link: 7,305, special: 1)
Number of created files: 194,890 (reg: 167,092, dir: 20,492, link: 7,305, special: 1)
Number of deleted files: 0
Number of regular files transferred: 167,036
Total file size: 6.68G bytes
Total transferred file size: 5.35G bytes
Literal data: 5.35G bytes
Matched data: 0 bytes
File list size: 5.64M
File list generation time: 0.021 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.36G
Total bytes received: 3.33M

sent 5.36G bytes  received 3.33M bytes  1.35M bytes/sec
total size is 6.68G  speedup is 1.25
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: true
## 2024-05-30_18-47_bookworm.img is 8913MiB with a root partition of 8397MiB.
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 17.09

I will try as before.... directly after reboot..........

UnconnectedBedna commented 3 months ago

As you can see, this is all very confusing. The only thing changed was instead of a 1 second sleep, it paused until you pressed enter.

There really is not much to do, I will try to up the sleep to 2 seconds. But I really have no clue, this makes 0 sense.

Please pull latest from testing and retry. I will keep thinking of what to do, but hopefully an even further sleep will be enough, 1s was enough on my systems...

sneumeister commented 3 months ago

Doing reboot...... and re-run script.... When waiting for [Enter] terminal 2:

 $ lsblk -o path,uuid /dev/loop0
PATH         UUID
/dev/loop0
/dev/loop0p1 3F92-1DB9
/dev/loop0p2 66378ef3-17b3-4f70-b069-2fa3e5b1f9c4

==> seems ok. Continue [Enter] will finisch w/o error.....

Number of files: 195,276 (reg: 167,473, dir: 20,497, link: 7,305, special: 1)
Number of created files: 194,893 (reg: 167,095, dir: 20,492, link: 7,305, special: 1)
Number of deleted files: 0
Number of regular files transferred: 167,039
Total file size: 6.68G bytes
Total transferred file size: 5.35G bytes
Literal data: 5.35G bytes
Matched data: 0 bytes
File list size: 5.57M
File list generation time: 0.037 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.36G
Total bytes received: 3.33M

sent 5.36G bytes  received 3.33M bytes  1.34M bytes/sec
total size is 6.68G  speedup is 1.25
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: true
## 2024-05-30_21-00_bookworm.img is 8943MiB with a root partition of 8427MiB.
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 12.54

Strange and not replaceable...

And even an update "-Ul" worked now... (directly startet after the backup above; no reboot or something)....?!?

$ sudo ./shrink-backup-f2fs -Ul 2024-05-30_21-00_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed NOT requested...
## Scanning filesystem and calculating...
## Looping img file...
##############################################################################
# Updating 2024-05-30_21-00_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# Resize operations not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: false
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: false
# Bootsector size: 515MiB
# Estemated root usage: 7893MiB
# Total img size: 8943MiB
##############################################################################
## Do you want to continue? [y/n] y

## Creating temp directory...
## Mounting img root partition...
## Mounting img boot partition...
## Backing up files...
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1016/1198)
dev/
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1007/1515)
etc/fake-hwclock.data
             20   0%    0.00kB/s    0:00:06 (xfr#1, ir-chk=1004/3288)
home/nodered/
             20   0%    0.00kB/s    0:00:20 (xfr#1, ir-chk=1220/8603)
deleting home/pi/smadata/SBFspot.db-wal
deleting home/pi/smadata/SBFspot.db-shm
             20   0%    0.00kB/s    0:00:23 (xfr#1, ir-chk=1562/9338)
home/nodered/.npm/_logs/
             20   0%    0.00kB/s    0:00:23 (xfr#1, ir-chk=1481/9338)

[...]

var/log/journal/c574c9ca1f484edf90ff5efca71c376e/user-1001.journal
        335.68M   4%  460.80kB/s    0:11:51 (xfr#21, to-chk=47/195284)
var/log/mosquitto/mosquitto.log
        335.75M   4%  460.88kB/s    0:11:51 (xfr#22, to-chk=31/195284)
var/log/openvpn/ipp.txt
        335.75M   4%  460.88kB/s    0:11:51 (xfr#23, to-chk=30/195284)
var/log/sbfspot.3/Neumi_de_20240530.log
        335.95M   4%  461.08kB/s    0:11:51 (xfr#24, to-chk=0/195284)

Number of files: 195,284 (reg: 167,481, dir: 20,497, link: 7,305, special: 1)
Number of created files: 10 (reg: 10)
Number of deleted files: 2 (reg: 2)
Number of regular files transferred: 24
Total file size: 6.89G bytes
Total transferred file size: 335.95M bytes
Literal data: 335.95M bytes
Matched data: 0 bytes
File list size: 524.26K
File list generation time: 0.023 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 340.73M
Total bytes received: 22.34K

sent 340.73M bytes  received 22.34K bytes  474.26K bytes/sec
total size is 6.89G  speedup is 20.22
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: false
## 2024-05-30_21-00_bookworm.img is 8943MiB
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 12.27

I have no idea.....

UnconnectedBedna commented 3 months ago

You never have to reboot. The reason I preassured you to do it was because your boot kept being unmounted and I wanted your system to be "clean" when starting the backup.

I have no idea either, but I increased the sleep timer to 2 seconds in the testing branch, hopefully that will keep it in check. :open_mouth:

sneumeister commented 3 months ago

Normally I do not want to reboot.... Using the same script from yesterday (*-f2fs) in update mode (which worked yesterday) today got error AND unmounted "/boot/firmware".

Here the full putty log when just opening the connection and do another trial.....

You see /boot/firmware mountpoint from lsblk. Then error of script and no mount anymore. Beside some cd and ls there were no other command; nevertheless the lost mountpoint..... strange....

$ login as: xxx
Authenticating with public key "xxx@george"
Linux george 6.6.28+rpt-rpi-v6 #1 Raspbian 1:6.6.28-1+rpt1 (2024-04-22) armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri May 31 11:27:39 2024 from 87.144.146.86
xxx@george:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 118.6G  0 part /
xxx@george:~ $ cd /mnt/netbackup/shrink-backup/
xxx@george:/mnt/netbackup/shrink-backup $ ls
2024-05-30_17-56_bookworm.img  2024-05-30_21-00_bookworm.img          exclude.txt    shrink-backup-f2fs
2024-05-30_18-47_bookworm.img  2024-05-30_shrink-backup_bookworm.img  shrink-backup  shrink-backup.log
xxx@george:/mnt/netbackup/shrink-backup $ ls -la
total 20779392
drwxr-xr-x 1 1024 users        366 May 30 21:00 .
drwxrwxrwx 1 root root         122 May 30 13:23 ..
-rw-r--r-- 1 1024 users 9401687552 May 30 18:01 2024-05-30_17-56_bookworm.img
-rw-r--r-- 1 1024 users 9346022912 May 30 20:17 2024-05-30_18-47_bookworm.img
-rw-r--r-- 1 1024 users 9377861120 May 30 22:28 2024-05-30_21-00_bookworm.img
-rw-r--r-- 1 1024 users 9415372288 May 30 16:21 2024-05-30_shrink-backup_bookworm.img
-rw-r--r-- 1 1024 users         74 May 20 22:27 exclude.txt
-rwxr-xr-x 1 1024 users      89349 May 30 17:30 shrink-backup
-rwxr-xr-x 1 1024 users      88925 May 30 18:26 shrink-backup-f2fs
-rw-r--r-- 1 1024 users      41660 May 30 22:28 shrink-backup.log
xxx@george:/mnt/netbackup/shrink-backup $ sudo ./shrink-backup-f2fs -Ulz 2024-05-30_21-00_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed requested...
## Scanning filesystem and calculating...
## Looping img file...
##############################################################################
# Updating 2024-05-30_21-00_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# Resize operations not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: true
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: false
# Bootsector size: 515MiB
# Estemated root usage: 8024MiB
# Total img size: 8943MiB
##############################################################################
## Do you want to continue? [y/n] y

## Creating temp directory...
## Mounting img root partition...
mount: /tmp/backup-AGR: wrong fs type, bad option, bad superblock on , missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
!! ROOT MOUNT FAILED!!!
## Cleanup function called with non zero exit code, something went wrong!!!
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 00.16
xxx@george:/mnt/netbackup/shrink-backup $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part
└─mmcblk0p2 179:2    0 118.6G  0 part /
xxx@george:/mnt/netbackup/shrink-backup $

OK and now it get's even stranger........ Just mounting /boot/firmware again and starting script the same way as abaove (shell-buffer Crsr-up) runs without problems now.............. Here exactly the commands performed (the first lines is same as the last from above):

## Elapsed time: 00.16
xxx@george:/mnt/netbackup/shrink-backup $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part
└─mmcblk0p2 179:2    0 118.6G  0 part /
xxx@george:/mnt/netbackup/shrink-backup $ 
xxx@george:/mnt/netbackup/shrink-backup $ sudo mount /boot/firmware
xxx@george:/mnt/netbackup/shrink-backup $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 118.6G  0 part /
xxx@george:/mnt/netbackup/shrink-backup $ 
xxx@george:/mnt/netbackup/shrink-backup $ sudo ./shrink-backup-f2fs -Ulz 2024-05-30_21-00_bookworm.img
## Debugging requested, writing to log file ./shrink-backup.log
## Zoom speed requested...
## Scanning filesystem and calculating...
## Looping img file...
##############################################################################
# Updating 2024-05-30_21-00_bookworm.img
# f2fs filesystem detected on root
# Autoexpand filesystem at boot not available for f2fs
# Resize operations not available for f2fs
# ----------------------------------------------------------------------------
# Write to logfile: true
# Zoom speed requested: true
# Autocalculate img root partition size: false
# Autoexpand filesystem at boot: false
# Use exclude.txt: false
# Bootsector size: 515MiB
# Estemated root usage: 8024MiB
# Total img size: 8943MiB
##############################################################################
## Do you want to continue? [y/n] y

## Creating temp directory...
## Mounting img root partition...
## Mounting img boot partition...
## Backing up files...
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1016/1198)
dev/
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1007/1515)
etc/fake-hwclock.data
             20   0%    0.00kB/s    0:00:19 (xfr#1, ir-chk=1562/9338)
home/pi/
home/pi/.bash_history
         46.66K   0%    2.39kB/s    0:00:19 (xfr#2, ir-chk=1456/9338)
home/pi/.lesshst
         46.70K   0%    2.39kB/s    0:00:19 (xfr#3, ir-chk=1477/9338)
home/pi/smadata/

[.....]

var/tmp/
        425.20M   6%  481.62kB/s    0:14:22 (xfr#70, to-chk=0/195310)

Number of files: 195,310 (reg: 167,507, dir: 20,497, link: 7,305, special: 1)
Number of created files: 26 (reg: 26)
Number of deleted files: 0
Number of regular files transferred: 70
Total file size: 7.03G bytes
Total transferred file size: 425.20M bytes
Literal data: 425.20M bytes
Matched data: 0 bytes
File list size: 524.26K
File list generation time: 0.024 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 430.01M
Total bytes received: 23.28K

sent 430.01M bytes  received 23.28K bytes  496.29K bytes/sec
total size is 7.03G  speedup is 16.34
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: false
## 2024-05-30_21-00_bookworm.img is 8943MiB
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 14.45

UnconnectedBedna commented 3 months ago

More things never crossed my mind users would do. :slightly_smiling_face: You even told me you were doing it, but I did not think it could create problems other than that the script in the same directory as the img file will not be backed up.. (the directory where the img file is created will automatically be excluded in the backup)

But:

xxx@george:/mnt/netbackup/shrink-backup $ sudo ./shrink-backup-f2fs -Ulz 2024-05-30_21-00_bookworm.img

You are not providing a ~~direct~~ absolute path for the img file, because you probably used tab and it finished it for you hence using a relative path? That might not work, because I never thought about anyone not providing an absolute path.. xD Not easy for you to know, but that might be the reason it failed the first time.

It also seems the script unmounted your boot, but I really find that hard to believe with all the checks being made ONLY to unount the temp directories created, if they do not exist or variables are empty, no unmounting should be done..

But you can check that in the log file you did not provide.. xD It should print out every single thing it does in the cleanup function when finnished or if error occur.

I updated the script on testing branch including a check if absolute paths are not being used, please clone and try exactly the way you tried before, NOT using ~~a direct~~ absolute path to the img file. :slightly_smiling_face:

sneumeister commented 3 months ago

More things never crossed my mind users would do. 🙂 You even told me you were doing it, but I did not think it could create problems other than that the script in the same directory as the img file will not be backed up.. (the directory where the img file is created will automatically be excluded in the backup)

Good to know... my thinking was about the .log file located in same directory as the script. Therefore I put script to another location; and since I wanted to backup to a NFS-share I placed into network folder under /net/netbackup . But since script expects exclude.txt in same directory I used sub-directory... since I already had sub-directory (where the backup .img's are I put it in there. And as you assumed, I am lazy in typing and so used the TAB key..... which consequently found the *.img in current directory and therefore absolute path was lost.... I did also not think about....

A nightmare of a tester... 😉

But you can check that in the log file you did not provide.. xD It should print out every single thing it does in the cleanup function when finnished or if error occur.

With the next trial.....

I updated the script on testing branch including a check if absolute paths are not being used, please clone and try exactly the way you tried before, NOT using ~a direct~ absolute path to the img file. 🙂

Okay, got new "testing" code. Backup is running; so fare it rsyncs.... I will come back with the result.

BTW. Adding a /tmp/* to exclude.txt should also do no harm? I did not find how this this folder is handled by raspbian (Debian) bookworm but I think you can not expect importent things there after reboot. (Some other distros do an regular timebased or reboot-based cleanup)

UnconnectedBedna commented 3 months ago

You did nothing wrong, it IS WAS a bug in the script for not warning you OR fixing these things for you, hopefully fixed now. :slightly_smiling_face:

A nightmare of a tester... 😉

HE** NO! You are an amazing QA tester, thank you for doing this with me!

Adding a /tmp/* to exclude.txt should also do no harm? I did not find how this this folder is handled by raspbian (Debian) bookworm but I think you can not expect importent things there after reboot. (Some other distros do an regular timebased or reboot-based cleanup)

Things might be a bit easier though if you read the documentation... :wink: https://github.com/UnconnectedBedna/shrink-backup?tab=readme-ov-file#excludetxt--t

The folder where the img file is created will ALWAYS be excluded in the backup. If -t option is selected, exclude.txt MUST exist (but can be empty) within the directory where the script is located or the script will exit with an error. Use one directory per line in exclude.txt. /directory/* = create directory but exclude content. /directory = exclude the directory completely.

If -t is NOT selected the following folders will be excluded:
/lost+found
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/swap

/tmp/* IS excluded both without the file OR if you use the default file provided on this gihub... :slightly_smiling_face:

sneumeister commented 3 months ago

Ok, Backup just finished w/o any issues. So I SSH'ed to my Raspberry (my home-Desktop, not the tablet I used the last days). /boot/firmware WAS mounted and IS after backup. xxx@george:/mnt/netbackup/shrink-backup $ sudo ./shrink-backup -Ulz 2024-05-30_21-00_bookworm.img

Number of files: 195,475 (reg: 167,645, dir: 20,524, link: 7,305, special: 1)
Number of created files: 27,482 (reg: 25,586, dir: 1,878, link: 18)
Number of deleted files: 27,317 (reg: 25,448, dir: 1,851, link: 18)
Number of regular files transferred: 27,152
Total file size: 6.91G bytes
Total transferred file size: 700.94M bytes
Literal data: 700.94M bytes
Matched data: 0 bytes
File list size: 917.46K
File list generation time: 0.033 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 706.92M
Total bytes received: 2.61M

sent 706.92M bytes  received 2.61M bytes  507.35K bytes/sec
total size is 6.91G  speedup is 9.74
## Rsync done...
## Please stand by...
## Backup done.
##############################################################################
## Write to logfile: true
## Autoexpand filesystem at boot: false
## Use exclude.txt: false
## /mnt/netbackup/shrink-backup/2024-05-30_21-00_bookworm.img is 8943MiB
##############################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 23.43

Log-File shrink-backup_2024-06-02.log

So far 👍 ....

UnconnectedBedna commented 3 months ago

Thank you again..

Please let me know how the updated backup went.

sneumeister commented 3 months ago

The final test is still missing.... 'dd' back to an SD card. Since the new 'bookworm' SD card is 128GB... only available free SD card is 64GB. Should not be a problem since last backup's size....

-rw-r--r-- 1 1024 users 9377861120 Jun  2 15:26 2024-05-30_21-00_bookworm.img

So let's see what comes out from my (buster...) Debian on the VirtualBox-Machine....

Hmmm..... doesn't look good..... ☹️

stephan@Vdebian32b:n$ sudo dd if=/mnt/netbackup/shrink-backup/2024-05-30_21-00_bookworm.img of=/dev/sdb status=progress 
607298048 bytes (607 MB, 579 MiB) copied, 419 s, 1,4 MB/s
[13870.582907] print_req_error: I/O error, dev sdb, sector 1188696 

dd: Schreiben in 1/dev/sdb': Eingabe-/Ausgabefehler 
1188697+0 Datensatze ein 
1188696+0 Datensatze aus 
608612352 bytes (609 MB, 580 MiB) copied, 423,027 s, 1,4 MB/s 
[13873.830302] print_req_error: I/O error, dev sdb, sector 1056808 
[13874.245878] print_req_error: I/O error, dev sdb, sector 8208 
[13874.246513] print_req_error: I/O error, dev sdb, sector 8232 
[13874.247106] print_req_error: I/O error, dev sdb, sector 1056840 
[13874.247734] print_req_error: I/O error, dev sdb, sector 1056904 
[13874.248319] print_req_error: I/O error, dev sdb, sector 1057032 
[13874.248887] print_req_error: I/O error, dev sdb, sector 1056808 
[13874.249446] Buffer I/O error on dev sdb2, logical block 40, async page read 
[13874.249970] Buffer I/O error on dev sdb2, logical block 41, async page read 
[13874.250510] Buffer I/O error on dev sdb2, logical block 42, async page read 
[13874.251014] Buffer I/O error on dev sdb2, logical block 43, async page read 
[13874.251527] Buffer I/O error on dev sdb2, logical block 44, async page read 
[13874.252000] Buffer I/O error on dev sdb2, logical block 45, async page read 
[13874.252636] Buffer I/O error on dev sdb2, logical block 46, async page read 
[13874.253140] Buffer I/O error on dev sdb2, logical block 47, async page read 
[13874.253618] print_req_error: I/O error, dev sdb, sector 8264 
[13874.254105] print_req_error: I/O error, dev sdb, sector 8328 
[13874.254561] Buffer I/O error on dev sdb1, logical block 2, async page read 
[13874.255064] Buffer I/O error on dev sdb1, logical block 2, async page read

And that card could also not start... start-up procedure got stuck with endlos error messages.....

But when writing the *.img with "Win32 Dsim Imager" directly from NAS (the nfs share is at the same time shared via SMB) to SD-Card showed no errors and I was able to boot from that SD and login.... no deeper tests but I am assuming -since booting works- everything else will also do. So I think the backup-image works.

Only the different behavior of writing SD card via "VirtualBox-Machine and dd" and via "Win32 Disk Imager" leaves some bitter taste....... 😕

UnconnectedBedna commented 3 months ago

Only the different behavior of writing SD card via "VirtualBox-Machine and dd" and via "Win32 Disk Imager" leaves some bitter taste....... 😕

Nothing to do with the img file. sdb is the device you write to, ie the sd-card.

So I think the backup-image works.

Great!

That means using f2fs works and I can close this.

The conversion will sooner or later be taken out of this script and moved to a separate application, but as for now, you can also convert ext4 to f2fs using the script.

Thanks for the help!