search

lzkelley / bkup_rpimage

Script to backup a Raspberry Pi disk image
GNU General Public License v3.0
162 stars 54 forks source link

Feature request: specify backup source from /dev/disk/by-id/ folder #14

Open ephestione opened 5 years ago

ephestione commented 5 years ago

Just like the title. I use USB boot, and this tool is awesome for backups, especially in the latest version where you can set image size and the PARTUUID's are automagically exported as well. Awesome for real. Now, since my raspi's work with several things hooked on USB, it MIGHT happen that the boot drive is sda today, and sdb tomorrow after I reboot, that's why whenever I need to refer to a drive, I prefer using the disk/partition identifiers in /dev/disk/by-id/ folder. With this script though, it seems disk identifiers under /dev/disk/ don't work, and I am forced to use /dev/sd*.

Is it a known limitation or can it be made so that they can be used to point to the source drive?

dolorosus commented 5 years ago

Did you try the "-i" option of the "start" command? E.g. bkup_rpimage start -c -i /dev/sda newbck.img ?

ephestione commented 5 years ago

Sorry, I didn't explain clearly :)

That is what I am doing currently, my currently working backup command is: ./backup start -ci /dev/sdb -s 3900 /path/to/destination/$(uname -n)-$(date +%Y-%m-%d).img (I renamed the script)

Recently though, it happened that sdb became sda, and it was obviously not working as the source and destination got mixed up with sda and sdb. Because of this problem with sdX not always being the same across several USB drives, this is what I would like to do instead: ./backup start -ci /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0\:0 -s 3900 /path/blah/test.img

but when I do that, the backup doesn't run, and this is the output:

Usage:
 blockdev -V
 blockdev --report [devices]
 blockdev [-v|-q] commands devices

Available commands:
 --getsz                   get size in 512-byte sectors
 --setro                   set read-only
 --setrw                   set read-write
 --getro                   get read-only
 --getdiscardzeroes        get discard zeroes support status
 --getss                   get logical block (sector) size
 --getpbsz                 get physical block (sector) size
 --getiomin                get minimum I/O size
 --getioopt                get optimal I/O size
 --getalignoff             get alignment offset in bytes
 --getmaxsect              get max sectors per request
 --getbsz                  get blocksize
 --setbsz <bytes>          set blocksize on file descriptor opening the block device
 --getsize                 get 32-bit sector count (deprecated, use --getsz)
 --getsize64               get size in bytes
 --setra <sectors>         set readahead
 --getra                   get readahead
 --setfra <sectors>        set filesystem readahead
 --getfra                  get filesystem readahead
 --flushbufs               flush buffers
 --rereadpt                reread partition table

Usage:
 blockdev -V
 blockdev --report [devices]
 blockdev [-v|-q] commands devices

Available commands:
 --getsz                   get size in 512-byte sectors
 --setro                   set read-only
 --setrw                   set read-write
 --getro                   get read-only
 --getdiscardzeroes        get discard zeroes support status
 --getss                   get logical block (sector) size
 --getpbsz                 get physical block (sector) size
 --getiomin                get minimum I/O size
 --getioopt                get optimal I/O size
 --getalignoff             get alignment offset in bytes
 --getmaxsect              get max sectors per request
 --getbsz                  get blocksize
 --setbsz <bytes>          set blocksize on file descriptor opening the block device
 --getsize                 get 32-bit sector count (deprecated, use --getsz)
 --getsize64               get size in bytes
 --setra <sectors>         set readahead
 --getra                   get readahead
 --setfra <sectors>        set filesystem readahead
 --getfra                  get filesystem readahead
 --flushbufs               flush buffers
 --rereadpt                reread partition table

Starting SD Image backup process
Creating sparse /path/blahblah/test.img, the apparent size of /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E
-0:0
0+0 records in
0+0 records out
0 bytes copied, 0.000795047 s, 0.0 kB/s
Attaching /path/blah/test.img to /dev/loop0
Copying partition table from /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0:0 to /dev/loop0
Checking that no-one is using this disk right now ... OK

Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x25089ae3

Old situation:

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> line 6: unsupported command

New situation:
Leaving.

Formatting partitions
mkfs.fat 4.1 (2017-01-24)
mkfs.vfat: unable to open /dev/loop0p1: No such file or directory
mke2fs 1.43.4 (31-Jan-2017)
The file /dev/loop0p2 does not exist and no size was specified.
e2fsck 1.43.4 (31-Jan-2017)
e2fsck: No such file or directory while trying to open /dev/loop0p2
Possibly non-existent device?
tune2fs 1.43.4 (31-Jan-2017)
tune2fs: option requires an argument -- 'U'
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-f] [-g group]
        [-i interval[d|m|w]] [-j] [-J journal_options] [-l]
        [-m reserved_blocks_percent] [-o [^]mount_options[,...]]
        [-p mmp_update_interval] [-r reserved_blocks_count] [-u user]
        [-C mount_count] [-L volume_label] [-M last_mounted_dir]
        [-O [^]feature[,...]] [-Q quota_options]
        [-E extended-option[,...]] [-T last_check_time] [-U UUID]
        [-I new_inode_size] [-z undo_file] device

Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x25089ae3

Command (m for help):
Expert command (m for help):
Enter the new disk identifier:
Disk identifier changed from 0x25089ae3 to 0x7b6fcbb3.

Expert command (m for help):
Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7b6fcbb3

Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Mounting /dev/loop01 and /dev/loop02 to /mnt/test.img/
mount: special device /dev/loop0p2 does not exist
mount: special device /dev/loop0p1 does not exist
Skipping rsync since /mnt/test.img/ is not a mount point
df: /dev/loop0p1: No such file or directory
df: /dev/loop0p2: No such file or directory
Flushing to disk
Unmounting /dev/loop01 and /dev/loop02 from /mnt/test.img/
umount: /mnt/test.img//boot: not mounted
umount: /mnt/test.img/: not mounted
rmdir: failed to remove '/mnt/test.img/': Directory not empty
Detaching /path/blah/test.img from /dev/loop0
partx: specified range <1:0> does not make sense
SD Image backup process completed.

Maybe there's another way to use a unique identifier for a disk, and NOT refer to it as sdX?

dolorosus commented 5 years ago

Are you shure you didn't changed the code? E.g. deleting line 39?

As this is not myrepo, I'm not able to create a testing branch. So as a test you may change line 39 from:

SDCARD=/dev/mmcblk0

to

SDCARD=/dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0\:0

and run the script without -i /dev...

ephestione commented 5 years ago

You got me! I acually had it commented out, but directly editing that line inside the script was something I had already tried. For the sake of actuality, I added that line back in, specifying the source from /dev/disk/by-id/, and removed the -i parameter from the commandline... still no go.

This is the output in this case (maybe the same as before):

Starting SD Image backup process
Creating sparse /blah/test.img, the apparent size of /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E
-0:0
0+0 records in
0+0 records out
0 bytes copied, 0.000760514 s, 0.0 kB/s
Attaching /blah/test.img to /dev/loop0
Copying partition table from /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0:0 to /dev/loop0
Checking that no-one is using this disk right now ... OK

Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x3a11ea83

Old situation:

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> line 6: unsupported command

New situation:
Leaving.

Formatting partitions
mkfs.fat 4.1 (2017-01-24)
mkfs.vfat: unable to open /dev/loop0p1: No such file or directory
mke2fs 1.43.4 (31-Jan-2017)
The file /dev/loop0p2 does not exist and no size was specified.
e2fsck 1.43.4 (31-Jan-2017)
e2fsck: No such file or directory while trying to open /dev/loop0p2
Possibly non-existent device?
tune2fs 1.43.4 (31-Jan-2017)
tune2fs: option requires an argument -- 'U'
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-f] [-g group]
        [-i interval[d|m|w]] [-j] [-J journal_options] [-l]
        [-m reserved_blocks_percent] [-o [^]mount_options[,...]]
        [-p mmp_update_interval] [-r reserved_blocks_count] [-u user]
        [-C mount_count] [-L volume_label] [-M last_mounted_dir]
        [-O [^]feature[,...]] [-Q quota_options]
        [-E extended-option[,...]] [-T last_check_time] [-U UUID]
        [-I new_inode_size] [-z undo_file] device

Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x3a11ea83

Command (m for help):
Expert command (m for help):
Enter the new disk identifier:
Disk identifier changed from 0x3a11ea83 to 0x7b6fcbb3.

Expert command (m for help):
Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7b6fcbb3

Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Mounting /dev/loop01 and /dev/loop02 to /mnt/test.img/
mount: special device /dev/loop0p2 does not exist
mount: special device /dev/loop0p1 does not exist
Skipping rsync since /mnt/test.img/ is not a mount point
df: /dev/loop0p1: No such file or directory
df: /dev/loop0p2: No such file or directory
Flushing to disk
Unmounting /dev/loop01 and /dev/loop02 from /mnt/test.img/
umount: /mnt/test.img//boot: not mounted
umount: /mnt/test.img/: not mounted
rmdir: failed to remove '/mnt/test.img/': Directory not empty
Detaching /blah/test.img from /dev/loop0
partx: specified range <1:0> does not make sense
SD Image backup process completed.
dolorosus commented 5 years ago

Looks like the size of /dev... could not be determined.

Can you please post the output of:

SDCARD=/dev/..... echo ${SIZE:-$(blockdev --getsz $SDCARD)} echo ${BLOCKSIZE:-$(blockdev --getss $SDCARD)}

ephestione commented 5 years ago

root@raspi3:/home/pi2pi# SDCARD=/dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0\:0 root@raspi3:/home/pi2pi# echo ${SIZE:-$(blockdev --getsz $SDCARD)} 60437492 root@raspi3:/home/pi2pi# echo ${BLOCKSIZE:-$(blockdev --getss $SDCARD)} 512

ephestione commented 5 years ago

It's notable though that the first time I did it, just like the output I pasted in the previous posts, it returned for both:

pi2pi@raspi3:~$ sudo su
root@raspi3:/home/pi2pi# echo ${SIZE:-$(blockdev --getsz $SDCARD)}

Usage:
 blockdev -V
 blockdev --report [devices]
 blockdev [-v|-q] commands devices

Available commands:
 --getsz                   get size in 512-byte sectors
 --setro                   set read-only
 --setrw                   set read-write
 --getro                   get read-only
 --getdiscardzeroes        get discard zeroes support status
 --getss                   get logical block (sector) size
 --getpbsz                 get physical block (sector) size
 --getiomin                get minimum I/O size
 --getioopt                get optimal I/O size
 --getalignoff             get alignment offset in bytes
 --getmaxsect              get max sectors per request
 --getbsz                  get blocksize
 --setbsz <bytes>          set blocksize on file descriptor opening the block device
 --getsize                 get 32-bit sector count (deprecated, use --getsz)
 --getsize64               get size in bytes
 --setra <sectors>         set readahead
 --getra                   get readahead
 --setfra <sectors>        set filesystem readahead
 --getfra                  get filesystem readahead
 --flushbufs               flush buffers
 --rereadpt                reread partition table
ephestione commented 5 years ago

I then tried, still after sudo su, to launch the command again, and:

Starting SD Image backup process
Creating sparse /media/giant/_PiBackup/test.img, the apparent size of /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E
-0:0
0+0 records in
0+0 records out
0 bytes copied, 0.00125749 s, 0.0 kB/s
Attaching /media/giant/_PiBackup/test.img to /dev/loop0
Copying partition table from /dev/disk/by-id/usb-Kingston_DataTraveler_3.0_08606E6B6446F351B726546E-0:0 to /dev/loop0
Checking that no-one is using this disk right now ... OK

Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xeece86b4

Old situation:

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> line 6: unsupported command

New situation:
Leaving.

Formatting partitions
mkfs.fat 4.1 (2017-01-24)
mkfs.vfat: unable to open /dev/loop0p1: No such file or directory
mke2fs 1.43.4 (31-Jan-2017)
The file /dev/loop0p2 does not exist and no size was specified.
e2fsck 1.43.4 (31-Jan-2017)
e2fsck: No such file or directory while trying to open /dev/loop0p2
Possibly non-existent device?
tune2fs 1.43.4 (31-Jan-2017)
tune2fs: option requires an argument -- 'U'
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-f] [-g group]
        [-i interval[d|m|w]] [-j] [-J journal_options] [-l]
        [-m reserved_blocks_percent] [-o [^]mount_options[,...]]
        [-p mmp_update_interval] [-r reserved_blocks_count] [-u user]
        [-C mount_count] [-L volume_label] [-M last_mounted_dir]
        [-O [^]feature[,...]] [-Q quota_options]
        [-E extended-option[,...]] [-T last_check_time] [-U UUID]
        [-I new_inode_size] [-z undo_file] device

Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xeece86b4

Command (m for help):
Expert command (m for help):
Enter the new disk identifier:
Disk identifier changed from 0xeece86b4 to 0x7b6fcbb3.

Expert command (m for help):
Command (m for help): Disk /dev/loop0: 3.8 GiB, 4089446400 bytes, 7987200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7b6fcbb3

Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Mounting /dev/loop01 and /dev/loop02 to /mnt/test.img/
mount: special device /dev/loop0p2 does not exist
mount: special device /dev/loop0p1 does not exist
Skipping rsync since /mnt/test.img/ is not a mount point
df: /dev/loop0p1: No such file or directory
df: /dev/loop0p2: No such file or directory
Flushing to disk
Unmounting /dev/loop01 and /dev/loop02 from /mnt/test.img/
umount: /mnt/test.img//boot: not mounted
umount: /mnt/test.img/: not mounted
rmdir: failed to remove '/mnt/test.img/': Directory not empty
Detaching /media/giant/_PiBackup/test.img from /dev/loop0
partx: specified range <1:0> does not make sense
SD Image backup process completed.
ephestione commented 5 years ago

I think I found the problem:

clone () {
    # cloning UUID and PARTUUID
    UUID=$(blkid -s UUID -o value ${SDCARD}p2)
    PTUUID=$(blkid -s PTUUID -o value ${SDCARD})
    e2fsck -f -y ${LOOPBACK}p2
    echo y|tune2fs ${LOOPBACK}p2 -U $UUID
    printf 'p\nx\ni\n%s\nr\np\nw\n' 0x${PTUUID}|fdisk "${LOOPBACK}"
    sync
}

In this subroutine the second partition is determined by adding "p2", while instead the SDCARD string is completed by either "-part1" or "-part2". Now that I know, I can edit it and try if that works, yet for a "clean" version it should be checked if the path is "/dev/[...]", or either "/dev/disk/by-id/[...]" and "/dev/disk/by-path/[...]" and add the partition suffix accordingly.

ephestione commented 5 years ago

Or rather, everywhere in the code the partition suffix is given as granted to be either "p1" or "p2", like

    mkfs.vfat -I ${LOOPBACK}p1
    mkfs.ext4 ${LOOPBACK}p2

so there would be the need to create another variable, SUFFIX, that is either "p" or "-part", and then do something like

${LOOPBACK}${SUFFIX}1

dolorosus commented 5 years ago

Thanks for this information. All references of the source partition are build this way. I will have a look if I can find a solution.

BUT the destination partition creation looks really strange. This complete part has to be rewritten. At the moment a correct patition table will be created by chance... ... I will fork this script to my repo an do a the rewrite.

ephestione commented 5 years ago

Since you would be doing an extensive rewrite of the code, and since @lzkelley doesn't appear to be mantaining this code at the moment, while you've been, you might even open another repo with his permission, say, dolorosus_rpimage 😄

dolorosus commented 5 years ago

Maybe...