Remote destination or image file?

[JustinFarris]

Forgive me if I'm barking up the wrong tree, or if I'm trying to invent a wheel that exists: I would like to have the option to make the destination of rpi-clone a file or a remote disk. I'm lazy and would love to have a way to schedule all the pis in my house to periodically backup to a fileserver.

[timdonovanuk]

+1

[jtmoderate876]

+1

[Paul-Reed]

+1

[GeminiServer]

+1

[steneor]

+1

[SeeSpotRun]

Should be easy enough to do a manual workaround (disclaimer: haven't tested yet):

$ truncate --size=8G /path/to/pibackup.img           # creates full size (but sparse) image
$                                                    # (choose size to be big enough
$                                                    # for source image)
$
$ sudo losetup -f --show /path/to/pibackup.img       # creates loop device
/dev/loop0
$                                                    # (take note of result, loop0 in this case)
$
$                                                    # (edit: probably need to select some
$                                                    # options regarding destination
$                                                    # partitioning on next step)
$ sudo rpi-clone loop0                               # clones to /path/to/temp.img
$                                                    # via loop device
$
$ sudo losetup --detach /dev/temp_loop               # clean up loop device
$
$ bzip2 /path/to/pibackup.img                        # (or use your preferred compression utility here)

If I get time I'll look at changing the rpi-clone code to add this as an option

Edit: hmmm, still needs a bit of work since partitions withing a loop device follow a different naming convention, ie first partition on /dev/loop0 id /dev/loop0p1. Will come back once I get it working.

[SeeSpotRun]

Ok first attempt at https://github.com/SeeSpotRun/rpi-clone/tree/image. Usage:

rpi-clone -i /path/to/image.img   # note: image can't already exist.

You can of course zip the resulting image.

Any willing beta testers?

[JustinFarris]

I'll try to give it a shot later today if nobody else gets to it.

[JustinFarris]

Quick clarification: If I'm reading the commit properly, it looks like this assumes you're writing to a "local" path, whether that path is actually a local disk or some mounted network storage device, correct? In other words, it would break if that destination path is an ssh/rsync-style "user@host:path", correct?

[jtmoderate876]

Gave @SeeSpotRun 's update a try copying to an image file on a mounted drive using: sudo ./rpi-clone -i "/mnt/netshare/rpi/image files/mine/rpi3 20180226 backup image.img" and I get an error regarding /dev/loop0

pi@rpi3:~/Downloads $ sudo ./rpi-clone -i "/mnt/netshare/rpi/image files/mine/rpi3 20180226 backup image.img" Error: /dev/loop0: unrecognised disk label

Booted disk: mmcblk0 15.8GB Destination disk: loop0 15.8GB

Part Size FS Label Part Size FS Label 1 /boot 43.5MB fat32 --
2 root 15.8GB ext4 rootfs

== Initialize: IMAGE mmcblk0 partition table to loop0 - FS types mismatch == 1 /boot (21.5MB used) : IMAGE to loop0p1 FSCK 2 root (4.7GB used) : RESIZE(15.8GB) MKFS SYNC to loop0p2

Run setup script : no Verbose mode : no -----------------------: WARNING : All destination disk loop0 data will be overwritten! : The partition structure will be imaged from mmcblk0. -----------------------:

Initialize and clone to the destination disk loop0? (yes/no):

[SeeSpotRun]

In other words, it would break if that destination path is an ssh/rsync-style "user@host:path", correct?

@JustinFarris that's correct, must be mounted. But you can use sshfs to mount your destination in your case.

[SeeSpotRun]

Error: /dev/loop0: unrecognised disk label

@jtmoderate876 it's going to be difficult to debug in bash; I'm going to write a stand-alone python script specifically for cloning a pi to an image.

[SolarDuck]

I have been a very satisfied user of rpi-clone for many years. Much kudos to BillW. The recent changes to Raspbian/stretch involving PARTUUID and SD card Disk identifier have necessitated a lot of work and I suppose this may have been one of the drivers for creation of rpi-clone2.

I have also recently been testing fsarchiver (http://www.fsarchiver.org/) as it incorporates a lot of things I think complement rpi-clone. It is more of an image backup tool rather than a file backup tool, which is how I view rpi-clone. The latest 0.8.4 version includes zstd compression. Only issue it does not address for me when I create an image file backing up /dev/mmcblk0p1 and /dev/mmcblk0p2 is the recent PARTUUID changes. It needs partitions created on the destination uSD card before recovery. These partitions can be any size large enough to hold the filesystems necessary to recover the volume of data involved.

I do not like incremental back-up systems for multi-generational back-ups and hence make a lot of use of fsarchiver's compression features, enhanced by its ability to simultaneously use all 4 of the RPi3 CPUs efficiently. The only thing image backup systems lack to my mind is the ability to selectively restore individual files. This has been addressed at https://ubuntuforums.org/showthread.php?t=1381103 for fsarchiver image files. This URL involves both a sparse file and loop device.

What I think would be great would be something like an rpi-clone bash script wrapper for fsarchiver to provide the best of both worlds.

[steneor]

works very well with: raspbian stretch -> hard drive ext3 mounted on /media/hdd raspbian stretch -> hard drive ext4 mounted on /media/hdd raspbian stretch -> Freebox remote cifs drive mounted on /media/freebox

The freebox is a box of the french internet access provider (free) Thank 's

[SeeSpotRun]

works very well with...

@steneor you mean https://github.com/SeeSpotRun/rpi-clone/tree/image works well? Did you test whether you can restore from the image to an sdcard which then boots ok on your RPi?

[steneor]

Yes, and it works

[SeeSpotRun]

Cool; doesn't work for me but it's a bug somewhere in rpi-clone, not in my patch. I think it's related to my sdcard having some extraneous partitions on it but I don't enjoy debugging bash so haven't dug into it as yet.

[SolarDuck]

I believe line 1670 of the modified code for writing to images, losetup -d "/dev/$dst_disk" Will not generate: losetup: /dev/loop0: failed to use device: No such device if changed to: losetup -d "/dev/$dst_disk"

with successful detaching of the loop device

[SeeSpotRun]

Thanks @SolarDuck. By the way you can comment directly in the code on github for example by clicking on the commit message an then the + symbol next to the line you want to comment against. See https://github.com/SeeSpotRun/rpi-clone/commit/f4e63b68c9f8abd57a6f2ef7d4c00b1967a308e9#diff-c82454179c2078b0c2eea6c60d92d0e7R1670

[marcert]

Hi, I tried out to save the img to filepath. But it does not really work and in iftop I see the following line: mount.exfat /dev/sda1 /media/pi/SSD -o rw,nodev,nosuid,uid=1000,gid=1000,iocharset=utf8,namecase=0,errors=remount-ro,uhelper=udisks2 with 100% usage of the interface. Who knows what happens there?

Thanks and Kind Regards, Marco

[SeeSpotRun]

Who knows what happens there?

It seems to be unable to mount /dev/sda1 as an exfat partition. Is it already mounted or are you able to mount it manually?

[marcert]

Hi, it seems that the problem comes for the SSD (connected to the Raspi via USB). If I try it with a good old USB harddisk it works. But I can mount the SSD and can also read and write files (mounted via the raspbian automount). It is a brandnew SSD maybe it has some problems... (although chkdsk on the PC said no problems at all). Has somebody another idea what I can try? Otherwise I will sent the SSD back.

Thanks and Kind Regards, Marco

[SeeSpotRun]

What is the output of:

lsblk -f /dev/sda1

?

[marcert]

The output is: sda1 exfat ssd CA62-D5DA /media/pi/ssd

Does this help?

Thanks and Kind Regards, Marco

[SeeSpotRun]

hmmm... can you try unmounting and then remounting as per the command rpi-clone was using:

$ umount /dev/sda1
$ mount.exfat /dev/sda1 /media/pi/SSD -o rw,nodev,nosuid,uid=1000,gid=1000,iocharset=utf8,namecase=0,errors=remount-ro,uhelper=udisks2

[marcert]

ahhh. After unmounting, creating the mountpoint manually, mounting I was able to create the img file on the external SSD. I only got the following if I answer yes for "initialize and clone to the destination disk loop0": `Initializing Imaging past the start of /boot partition 2. => dd if=/dev/mmcblk0 of=/dev/loop0 bs=1M count=50 ... Resizing last partition to end of disk ... Resize success. Changing destination Disk ID ...Re-reading the partition table failed.: Invalid argument

Delaying so partprobe can update /dev entries ... => fsck -p /dev/loop0p1 ... => mkfs -t ext4 -L dex99 /dev/loop0p2 ...

Syncing file systems (can take a long time) Syncing mounted partitions: Mounting /dev/loop0p2 on /mnt/clone => rsync // /mnt/clone with-root-excludes ...rsync: readlink_stat("/home/pi/thinclient_drives") failed: Permission denied (13) IO error encountered -- skipping file deletion rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2] ` But the img-file is there. If I answer "no" I get the message "Aborting!" and also an img file. Now I will test if these img files are working...

Thanks and Kind Regards, Marco

[PieMuncher]

Another +1 for this in rpi-clone. Cloning to a NAS would be v convenient!

Going to to approach suggested above... looks like it should be fairly easy to include in rpi-clone - have you/would you submit a pull request @SeeSpotRun ?

[SeeSpotRun]

I haven't because I feel it still needs a little debugging and I have no appetite for debugging in bash. But I'm more than happy for someone else to take what I did, firm it up a little, do some testing and submit a PR.

[PieMuncher]

Your approach worked well for me @SeeSpotRun - would be nice if it was built in - I hope the contributors note the number of people wanting this feature :) I also found another project on github doing this called clonepi.

[CaptainMidnight]

Hi what's the status of this requested functionality as these comments are over 1.5yrs old?

Is there any timeline available where this cloning to a compressed image file may be available?

[p1r473]

@SeeSpotRun would love to see your image creation as a pull request here.