Fixes to wyng-extract.sh

tlaurion commented 3 years ago

@tasket, here are some typo fixes (grep -q -> grep -E, none -> None) that were making this script non-working (tested over Fedora-32 live)

the "lv -> lvm lv" fix here will permit to call the script without modification under Heads (which doesn't have the symlinks to call lvm subcalls). Calling "lvm lv" programs don't break it for other systems.

That worked into restoring root-autosnap under root in my test for bzip2 archives! Yessss!

diff --git a/misc/wyng-extract.sh b/misc/wyng-extract.sh
index cb2fb43..3ef95bc 100755
--- a/misc/wyng-extract.sh
+++ b/misc/wyng-extract.sh
@@ -58,7 +58,7 @@ rm -rf $tmpdir  &&  mkdir $tmpdir
 ( cd "$voldir";  curdir=`pwd`

   # Check that format version is 1 or 2
-  arch_ver=`grep -q '^format_ver =' volinfo | awk '{print $3}'`
+  arch_ver=`grep -E '^format_ver =' volinfo | awk '{print $3}'`
   case $arch_ver in
     1|2)  format_ver=$arch_ver;;
     *)    echo "Error: Did not find a compatible format.";  exit 1;;
@@ -71,7 +71,7 @@ rm -rf $tmpdir  &&  mkdir $tmpdir
   # Add session column to manifests and create symlinks using sequence number.
   if [ -z "$sestag" ]; then sestag=${s_last:2}; fi
   session=$s_last
-  while [ ! "$session" = 'none' ]; do
+  while [ ! "$session" = 'None' ]; do
     ln=`grep '^previous =' $session/info`;  read one two s_prev <<<"$ln"
     if [ -z "$sesnames" ] && [ ! "${session:2}" = "$sestag" ]; then session=$s_prev;  continue; fi

@@ -218,9 +218,9 @@ if [ -n "$opt_list" ]; then exit 0; fi
   if [ ! -b "$outvol" ]; then
     if [ -z "$opt_sparse" ]; then rm "$outvol"; fi
     truncate --size $volsize "$outvol"
-  elif lvdisplay "$outvol" >/dev/null; then
+  elif lvm lvdisplay "$outvol" >/dev/null; then
     if [ -z "$opt_sparse" ]; then blkdiscard "$outvol"; fi
-    lvresize -L ${volsize}B "$outvol" 2>/dev/null || true
+    lvm lvresize -L ${volsize}B "$outvol" 2>/dev/null || true
   fi
   if [ ! -e "$outvol" ]; then
     echo "Error: Output/save path does not exist!"

tlaurion commented 3 years ago

Ran some tests with two crafted scripts. One using python and the other using shell, both ran over Fedora-32 Live ISO.

user@Insurgo:/media/user/9e2db9c8-f89b-4f62-9aa2-d18ba800afd8$ cat wyng-extract-automatized.sh

#!/bin/bash
WYNG_BACKUPS_ARCHIVE_DIR=/media/wyng-backups/home/user/wyng-backups/wyng.backup/
QUBES_VM_POOL="/dev/qubes_dom0"

OEM_MOUNT="/media/wyng-backups/home/user/"
GPG_OEM_KEY="/media/wyng-backups/home/user/Insurgo_2022-06-15.asc"

#Under Q4.0, VMs and dom0 root LVMs are under the same pool. On Q4.1, they are on different pools
QUBES_DOM0_POOL="/dev/qubes_dom0"

#Depending on which system we are restoring from, we might or not need sudo. Put to "" if unneeded.
SUDO_NEEDED="sudo"

#Following exaple takes into consideration Q4.0 OEM disk image.
if ! $SUDO_NEEDED mount | grep -q wyng-backups;then
    $SUDO_NEEDED cryptsetup luksOpen /dev/sda2 qubes_dom0
    $SUDO_NEEDED lvm vgchange -ay qubes_dom0
    $SUDO_NEEDED lvm lvchange -ay qubes_dom0
    $SUDO_NEEDED mkdir -p /media/wyng-backups
    $SUDO_NEEDED mount /dev/qubes_dom0/vm-wyng-backups-private /media/wyng-backups
fi

gpg --import "$GPG_OEM_KEY" > /dev/null 2>&1
cd "$OEM_MOUNT" > /dev/null 2>&1
./verify_backups.sh
cd - > /dev/null 2>&1

read -p "Press Enter to continue or CTRL-C to abort..."

RestorableLVMs=$(find "$WYNG_BACKUPS_ARCHIVE_DIR/default/" -maxdepth 1 -type d | awk -F "/" '{print $NF}')
echo "$RestorableLVMs" | while read lvm; do 
    if [ -n "$lvm" ]; then
        #dom0 root is backuped under root-autosnap LVM from dom0's systemd's shutdown script.
        if [ "$lvm" == "root-autosnap" ]; then
            dest="$QUBES_DOM0_POOL/root"
        else
            dest="$QUBES_VM_POOL/$lvm"
        fi

        echo "Restoring wyng-backup archive: $lvm into Thin LVM volume: $dest..."   
        time $SUDO_NEEDED ./wyng-extract.sh -s -o $dest $WYNG_BACKUPS_ARCHIVE_DIR $lvm 
    fi
done

user@Insurgo:/media/user/9e2db9c8-f89b-4f62-9aa2-d18ba800afd8$ cat wyng-python-automatized.sh

#!/bin/bash

#
#GENERAL PATHS
WYNG_BACKUPS_ARCHIVE_DIR="/media/wyng-backups/home/user/wyng-backups/wyng.backup"
QUBES_VM_POOL="/dev/qubes_dom0"
OEM_MOUNT="/media/wyng-backups/home/user/"
GPG_OEM_KEY="/media/wyng-backups/home/user/Insurgo_2022-06-15.asc"

#Under Q4.0, VMs and dom0 root LVMs are under the same pool. On Q4.1, they are on different pools
QUBES_DOM0_POOL="/dev/qubes_dom0"

#Depending on which system we are restoring from, we might or not need sudo. Put to "" if unneeded.
SUDO_NEEDED="sudo"

#WYNG python version options
SPARSE_MODE="--sparse-write"
WYNG_BACKUP_FROM_ARCHIVE_DIR="media/wyng-backups/home/user/wyng-backups"

#Following exaple takes into consideration Q4.0 OEM disk image.
if ! $SUDO_NEEDED mount | grep -q wyng-backups;then
    $SUDO_NEEDED cryptsetup luksOpen /dev/sda2 qubes_dom0
    $SUDO_NEEDED lvm vgchange -ay qubes_dom0
    $SUDO_NEEDED lvm lvchange -ay qubes_dom0
    $SUDO_NEEDED mkdir -p /media/wyng-backups
    $SUDO_NEEDED mount /dev/qubes_dom0/vm-wyng-backups-private /media/wyng-backups
fi

gpg --import "$GPG_OEM_KEY" > /dev/null 2>&1
cd "$OEM_MOUNT" > /dev/null 2>&1
./verify_backups.sh

cd - > /dev/null 2>&1

read -p "Press Enter to continue or CTRL-C to abort..."

RestorableLVMs=$(find "$WYNG_BACKUPS_ARCHIVE_DIR/default/" -maxdepth 1 -type d | awk -F "/" '{print $NF}')
echo "$RestorableLVMs" | while read lvm; do 
    if [ -n "$lvm" ]; then
        #dom0 root is backuped under root-autosnap LVM from dom0's systemd's shutdown script.
        if [ "$lvm" == "root-autosnap" ]; then
            dest="$QUBES_DOM0_POOL/root"
        else
            dest="$QUBES_VM_POOL/$lvm"
        fi

        echo ""
        echo "Restoring wyng-backup archive: $lvm into Thin LVM volume: $dest..."
        echo "Running command: time $SUDO_NEEDED ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=$WYNG_BACKUP_FROM_ARCHIVE_DIR receive $lvm --save-to $dest $SPARSE_MODE"    
        time $SUDO_NEEDED ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=$WYNG_BACKUP_FROM_ARCHIVE_DIR receive $lvm --save-to $dest $SPARSE_MODE 
    fi
done

The python script results into a totally functional system (still /boot is not restored for the moment).
The wyng-extract.sh script (as modified in OP) results in all sha256sum -c not being valid for some reason (fails receiving volumes)

Here are the outputs:

[liveuser@localhost-live isoscan]$ time ./wyng-extract-automatized.sh
gpg: assuming signed data in 'backups.sha256sum'
gpg: Signature made Sat 31 Jul 2021 09:59:27 AM EDT
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
Backup signature is VALID

Public key of insugo@riseup.net at the time of this writing was:
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
This should match the public key fingerprint seen above from previous signature verification.
This public key can be downloaded/verified against downloadable links provided under GPG section at https://insurgo.ca#contact

Integrity of files checked in manifest are intact!
All good!
Press Enter to continue or CTRL-C to abort...
Restoring wyng-backup archive: vm-sys-firewall-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-firewall-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-sys-firewall-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.563s
user    0m14.672s
sys 0m4.271s
Restoring wyng-backup archive: vm-sys-usb-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-usb-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-sys-usb-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.449s
user    0m14.631s
sys 0m4.246s
Restoring wyng-backup archive: vm-work-private into Thin LVM volume: /dev/qubes_dom0/vm-work-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-work-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.343s
user    0m14.541s
sys 0m4.196s
Restoring wyng-backup archive: vm-fedora-33-root into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-fedora-33-root.
Volume size = 21474836480 bytes.
Hashing volume 21206401024 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    5m51.416s
user    9m42.970s
sys 1m2.371s
Restoring wyng-backup archive: vm-sys-net-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-net-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-sys-net-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.456s
user    0m14.655s
sys 0m4.215s
Restoring wyng-backup archive: vm-windows-10-standalone-root into Thin LVM volume: /dev/qubes_dom0/vm-windows-10-standalone-root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-windows-10-standalone-root.
Volume size = 32212254720 bytes.
Hashing volume 31943819264 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    27m8.285s
user    48m25.737s
sys 2m38.775s
Restoring wyng-backup archive: vm-anon-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-anon-whonix-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-anon-whonix-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.454s
user    0m14.620s
sys 0m4.253s
Restoring wyng-backup archive: vm-fedora-33-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-fedora-33-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.531s
user    0m14.785s
sys 0m4.194s
Restoring wyng-backup archive: vm-vault-private into Thin LVM volume: /dev/qubes_dom0/vm-vault-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-vault-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.412s
user    0m14.605s
sys 0m4.221s
Restoring wyng-backup archive: vm-whonix-ws-15-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-dvm-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-whonix-ws-15-dvm-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.355s
user    0m14.507s
sys 0m4.201s
Restoring wyng-backup archive: vm-sys-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-whonix-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-sys-whonix-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.397s
user    0m14.555s
sys 0m4.221s
Restoring wyng-backup archive: vm-whonix-ws-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-whonix-ws-15-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.393s
user    0m14.527s
sys 0m4.306s
Restoring wyng-backup archive: vm-fedora-33-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-dvm-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-fedora-33-dvm-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m15.741s
user    0m24.156s
sys 0m4.549s
Restoring wyng-backup archive: vm-default-mgmt-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-default-mgmt-dvm-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-default-mgmt-dvm-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.549s
user    0m14.569s
sys 0m4.288s
Restoring wyng-backup archive: vm-debian-10-root into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-debian-10-root.
Volume size = 21474836480 bytes.
Hashing volume 21206401024 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    8m47.503s
user    15m2.643s
sys 1m13.370s
Restoring wyng-backup archive: vm-windows-10-standalone-private into Thin LVM volume: /dev/qubes_dom0/vm-windows-10-standalone-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-windows-10-standalone-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.303s
user    0m14.420s
sys 0m4.271s
Restoring wyng-backup archive: vm-debian-10-private into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-debian-10-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.513s
user    0m14.624s
sys 0m4.384s
Restoring wyng-backup archive: vm-whonix-ws-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-whonix-ws-15-root.
Volume size = 21474836480 bytes.
Hashing volume 21206401024 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    4m46.302s
user    7m50.943s
sys 0m55.797s
Restoring wyng-backup archive: vm-whonix-gw-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-whonix-gw-15-root.
Volume size = 21474836480 bytes.
Hashing volume 21206401024 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    3m45.298s
user    6m2.039s
sys 0m51.411s
Restoring wyng-backup archive: vm-untrusted-private into Thin LVM volume: /dev/qubes_dom0/vm-untrusted-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-untrusted-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.457s
user    0m14.551s
sys 0m4.288s
Restoring wyng-backup archive: vm-whonix-gw-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-whonix-gw-15-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.394s
user    0m14.530s
sys 0m4.278s
Restoring wyng-backup archive: root-autosnap into Thin LVM volume: /dev/qubes_dom0/root...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume root-autosnap.
Volume size = 190551425024 bytes.
Hashing volume 190320738304 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    18m24.188s
user    26m52.361s
sys 6m31.952s
Restoring wyng-backup archive: vm-wyng-rsync_net-Insurgo-private into Thin LVM volume: /dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-wyng-rsync_net-Insurgo-private.
Volume size = 24696061952 bytes.
Hashing volume 24427626496 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    2m2.552s
user    2m52.834s
sys 0m49.442s
Restoring wyng-backup archive: vm-personal-private into Thin LVM volume: /dev/qubes_dom0/vm-personal-private...
Wyng archive extractor, V0.3.0 20210605
Getting metadata for volume vm-personal-private.
Volume size = 2147483648 bytes.
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.407s
user    0m14.559s
sys 0m4.240s

real    73m51.567s
user    121m7.752s
sys 15m15.811s

[liveuser@localhost-live isoscan]$ time ./wyng-python-automatized.sh
gpg: assuming signed data in 'backups.sha256sum'
gpg: Signature made Sat 31 Jul 2021 09:59:27 AM EDT
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
Backup signature is VALID

Public key of insugo@riseup.net at the time of this writing was:
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
This should match the public key fingerprint seen above from previous signature verification.
This public key can be downloaded/verified against downloadable links provided under GPG section at https://insurgo.ca#contact

Integrity of files checked in manifest are intact!
All good!
Press Enter to continue or CTRL-C to abort...

Restoring wyng-backup archive: vm-sys-firewall-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-firewall-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-firewall-private --save-to /dev/qubes_dom0/vm-sys-firewall-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-firewall-private 20210712-130750
Saving to logical volume '/dev/qubes_dom0/vm-sys-firewall-private'
100.00%

real    0m1.390s
user    0m0.469s
sys 0m0.803s

Restoring wyng-backup archive: vm-sys-usb-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-usb-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-usb-private --save-to /dev/qubes_dom0/vm-sys-usb-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-usb-private 20210712-130750
Saving to logical volume '/dev/qubes_dom0/vm-sys-usb-private'
100.00%

real    0m1.380s
user    0m0.475s
sys 0m0.803s

Restoring wyng-backup archive: vm-work-private into Thin LVM volume: /dev/qubes_dom0/vm-work-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-work-private --save-to /dev/qubes_dom0/vm-work-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-work-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-work-private'
100.00%

real    0m1.360s
user    0m0.475s
sys 0m0.794s

Restoring wyng-backup archive: vm-fedora-33-root into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-root --save-to /dev/qubes_dom0/vm-fedora-33-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-root 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-root'
100.00%
Diff bytes: 4282908672

real    3m57.212s
user    3m2.177s
sys 0m14.167s

Restoring wyng-backup archive: vm-sys-net-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-net-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-net-private --save-to /dev/qubes_dom0/vm-sys-net-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-net-private 20210712-130750
Saving to logical volume '/dev/qubes_dom0/vm-sys-net-private'
100.00%

real    0m1.570s
user    0m0.486s
sys 0m0.847s

Restoring wyng-backup archive: vm-windows-10-standalone-root into Thin LVM volume: /dev/qubes_dom0/vm-windows-10-standalone-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-windows-10-standalone-root --save-to /dev/qubes_dom0/vm-windows-10-standalone-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-windows-10-standalone-root 20210730-184147
Saving to logical volume '/dev/qubes_dom0/vm-windows-10-standalone-root'
100.00%

real    21m13.677s
user    20m8.014s
sys 0m44.035s

Restoring wyng-backup archive: vm-anon-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-anon-whonix-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-anon-whonix-private --save-to /dev/qubes_dom0/vm-anon-whonix-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-anon-whonix-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-anon-whonix-private'
100.00%

real    0m1.449s
user    0m0.453s
sys 0m0.818s

Restoring wyng-backup archive: vm-fedora-33-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-private --save-to /dev/qubes_dom0/vm-fedora-33-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-private'
100.00%

real    0m1.391s
user    0m0.495s
sys 0m0.790s

Restoring wyng-backup archive: vm-vault-private into Thin LVM volume: /dev/qubes_dom0/vm-vault-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-vault-private --save-to /dev/qubes_dom0/vm-vault-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-vault-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-vault-private'
100.00%

real    0m1.411s
user    0m0.491s
sys 0m0.808s

Restoring wyng-backup archive: vm-whonix-ws-15-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-dvm-private --save-to /dev/qubes_dom0/vm-whonix-ws-15-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-dvm-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-dvm-private'
100.00%

real    0m1.400s
user    0m0.454s
sys 0m0.802s

Restoring wyng-backup archive: vm-sys-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-whonix-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-whonix-private --save-to /dev/qubes_dom0/vm-sys-whonix-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-whonix-private 20210712-130750
Saving to logical volume '/dev/qubes_dom0/vm-sys-whonix-private'
100.00%

real    0m1.424s
user    0m0.474s
sys 0m0.807s

Restoring wyng-backup archive: vm-whonix-ws-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-private --save-to /dev/qubes_dom0/vm-whonix-ws-15-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-private'
100.00%

real    0m1.392s
user    0m0.439s
sys 0m0.836s

Restoring wyng-backup archive: vm-fedora-33-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-dvm-private --save-to /dev/qubes_dom0/vm-fedora-33-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-dvm-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-dvm-private'
100.00%

real    0m5.430s
user    0m4.203s
sys 0m0.985s

Restoring wyng-backup archive: vm-default-mgmt-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-default-mgmt-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-default-mgmt-dvm-private --save-to /dev/qubes_dom0/vm-default-mgmt-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-default-mgmt-dvm-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-default-mgmt-dvm-private'
100.00%

real    0m1.629s
user    0m0.445s
sys 0m0.890s

Restoring wyng-backup archive: vm-debian-10-root into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-debian-10-root --save-to /dev/qubes_dom0/vm-debian-10-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-debian-10-root 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-debian-10-root'
100.00%

real    6m7.951s
user    5m37.795s
sys 0m18.488s

Restoring wyng-backup archive: vm-windows-10-standalone-private into Thin LVM volume: /dev/qubes_dom0/vm-windows-10-standalone-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-windows-10-standalone-private --save-to /dev/qubes_dom0/vm-windows-10-standalone-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-windows-10-standalone-private 20210720-104150
Saving to logical volume '/dev/qubes_dom0/vm-windows-10-standalone-private'
100.00%

real    0m1.382s
user    0m0.455s
sys 0m0.806s

Restoring wyng-backup archive: vm-debian-10-private into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-debian-10-private --save-to /dev/qubes_dom0/vm-debian-10-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-debian-10-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-debian-10-private'
100.00%

real    0m1.375s
user    0m0.437s
sys 0m0.844s

Restoring wyng-backup archive: vm-whonix-ws-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-root --save-to /dev/qubes_dom0/vm-whonix-ws-15-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-root 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-root'
100.00%

real    2m41.541s
user    2m24.035s
sys 0m11.768s

Restoring wyng-backup archive: vm-whonix-gw-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-gw-15-root --save-to /dev/qubes_dom0/vm-whonix-gw-15-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-gw-15-root 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-whonix-gw-15-root'
100.00%

real    1m49.088s
user    1m34.702s
sys 0m10.030s

Restoring wyng-backup archive: vm-untrusted-private into Thin LVM volume: /dev/qubes_dom0/vm-untrusted-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-untrusted-private --save-to /dev/qubes_dom0/vm-untrusted-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-untrusted-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-untrusted-private'
100.00%

real    0m1.385s
user    0m0.424s
sys 0m0.817s

Restoring wyng-backup archive: vm-whonix-gw-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-gw-15-private --save-to /dev/qubes_dom0/vm-whonix-gw-15-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-gw-15-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-whonix-gw-15-private'
100.00%

real    0m1.479s
user    0m0.438s
sys 0m0.850s

Restoring wyng-backup archive: root-autosnap into Thin LVM volume: /dev/qubes_dom0/root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive root-autosnap --save-to /dev/qubes_dom0/root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : root-autosnap 20210730-222941
Saving to logical volume '/dev/qubes_dom0/root'
100.00%

real    4m3.823s
user    2m45.477s
sys 1m6.326s

Restoring wyng-backup archive: vm-wyng-rsync_net-Insurgo-private into Thin LVM volume: /dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-wyng-rsync_net-Insurgo-private --save-to /dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-wyng-rsync_net-Insurgo-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private'
100.00%

real    0m13.724s
user    0m4.855s
sys 0m8.271s

Restoring wyng-backup archive: vm-personal-private into Thin LVM volume: /dev/qubes_dom0/vm-personal-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-personal-private --save-to /dev/qubes_dom0/vm-personal-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-personal-private 20210730-112039
Saving to logical volume '/dev/qubes_dom0/vm-personal-private'
100.00%

real    0m1.466s
user    0m0.458s
sys 0m0.848s

real    40m36.589s
user    35m49.363s
sys 3m7.284s

tasket commented 3 years ago

@tlaurion Thanks. The diff in initial comment needs some adjustment, however. Looking at my own archives there is a chance that "none" lowercase may appear, so that test should be case insensitive (this is already handled in Wyng). Also the -E option for grep would seem to be superfluous in that spot.

I'll review your other comments in a bit.

tasket commented 3 years ago

@tlaurion I can't say yet what's causing the error. If its related to the 'while' loop, then changing it to this might fix it (notice that 'none' is lowercase):

  while [ ! ${session,,} = 'none' ]; do

Posting a sample of the data in '/tmp/wyng-extract/diff-manifest' would be the best place to start analyzing.

tasket commented 3 years ago

FYI there does seem to be a problem when -s is used and the local/archive vol sizes don't match. I'll post an update soon to handle that situation.

tasket commented 3 years ago

@tlaurion Updated.

tlaurion commented 3 years ago

@tasket https://github.com/tasket/wyng-backup/commit/a81e7c04ac6a6d8ccd1c18898f5e0fc3c24400d7 introduces a new output error, stating for example, that dm group dm-32 was not found. The rest of the output above is the same.

The restoration with wyng python script works as intended with automation scripts above unchanged.

tlaurion commented 3 years ago

@tasket sorry for the delay.

Exposing what you asked before with a bit more details, showing fix03 branch (python) works vs wyng-extract.sh from latest commit https://github.com/tasket/wyng-backup/commit/a81e7c04ac6a6d8ccd1c18898f5e0fc3c24400d7. Disregard the first lines of output, just wanted to see output of valid detached signed digest with a voume having been updated without resigning.

fix03 python automatized receive (from script above):

[liveuser@localhost-live isoscan]$ ./wyng-python-automatized.sh 
gpg: assuming signed data in 'backups.sha256sum'
gpg: Signature made Tue 10 Aug 2021 09:26:41 AM EDT
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
Backup signature is VALID

Public key of insugo@riseup.net at the time of this writing was:
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
This should match the public key fingerprint seen above from previous signature verification.
This public key can be downloaded/verified against downloadable links provided under GPG section at https://insurgo.ca#contact

./boot.bzip2: FAILED
./wyng-backups/wyng.backup/default/root-autosnap/volinfo: FAILED
sha256sum: ./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/info: No such file or directory
./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/info: FAILED open or read
sha256sum: ./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/manifest: No such file or directory
./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/manifest: FAILED open or read
sha256sum: WARNING: 2 listed files could not be read
sha256sum: WARNING: 2 computed checksums did NOT match
INTEGRITY OF BACKUP MANIFEST WAS TAMPERED WITH!
Press Enter to continue or CTRL-C to abort...

Restoring wyng-backup archive: vm-windows10_office365proplus-root into Thin LVM volume: /dev/qubes_dom0/vm-windows10_office365proplus-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-windows10_office365proplus-root --save-to /dev/qubes_dom0/vm-windows10_office365proplus-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-windows10_office365proplus-root 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-windows10_office365proplus-root'
100.00%

real    14m6.179s
user    13m18.529s
sys 0m33.308s

Restoring wyng-backup archive: vm-sys-firewall-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-firewall-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-firewall-private --save-to /dev/qubes_dom0/vm-sys-firewall-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-firewall-private 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-sys-firewall-private'
100.00%

real    0m1.596s
user    0m0.525s
sys 0m0.771s

Restoring wyng-backup archive: vm-sys-usb-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-usb-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-usb-private --save-to /dev/qubes_dom0/vm-sys-usb-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-usb-private 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-sys-usb-private'
100.00%
Diff bytes: 1835008

real    0m1.464s
user    0m0.493s
sys 0m0.822s

Restoring wyng-backup archive: vm-work-private into Thin LVM volume: /dev/qubes_dom0/vm-work-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-work-private --save-to /dev/qubes_dom0/vm-work-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-work-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-work-private'
100.00%

real    0m1.343s
user    0m0.461s
sys 0m0.747s

Restoring wyng-backup archive: vm-fedora-33-root into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-root --save-to /dev/qubes_dom0/vm-fedora-33-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-root 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-root'
100.00%

real    3m28.124s
user    3m9.546s
sys 0m12.930s

Restoring wyng-backup archive: vm-sys-net-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-net-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-net-private --save-to /dev/qubes_dom0/vm-sys-net-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-net-private 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-sys-net-private'
100.00%

real    0m1.465s
user    0m0.465s
sys 0m0.854s

Restoring wyng-backup archive: vm-anon-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-anon-whonix-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-anon-whonix-private --save-to /dev/qubes_dom0/vm-anon-whonix-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-anon-whonix-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-anon-whonix-private'
100.00%

real    0m1.361s
user    0m0.469s
sys 0m0.798s

Restoring wyng-backup archive: vm-fedora-33-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-private --save-to /dev/qubes_dom0/vm-fedora-33-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-private'
100.00%

real    0m1.459s
user    0m0.520s
sys 0m0.800s

Restoring wyng-backup archive: vm-vault-private into Thin LVM volume: /dev/qubes_dom0/vm-vault-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-vault-private --save-to /dev/qubes_dom0/vm-vault-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-vault-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-vault-private'
100.00%

real    0m1.528s
user    0m0.521s
sys 0m0.826s

Restoring wyng-backup archive: vm-whonix-ws-15-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-dvm-private --save-to /dev/qubes_dom0/vm-whonix-ws-15-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-dvm-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-dvm-private'
100.00%

real    0m1.409s
user    0m0.488s
sys 0m0.780s

Restoring wyng-backup archive: vm-sys-whonix-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-whonix-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-sys-whonix-private --save-to /dev/qubes_dom0/vm-sys-whonix-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-sys-whonix-private 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-sys-whonix-private'
100.00%

real    0m1.422s
user    0m0.471s
sys 0m0.818s

Restoring wyng-backup archive: vm-whonix-ws-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-private --save-to /dev/qubes_dom0/vm-whonix-ws-15-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-private'
100.00%

real    0m1.383s
user    0m0.459s
sys 0m0.792s

Restoring wyng-backup archive: vm-fedora-33-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-fedora-33-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-fedora-33-dvm-private --save-to /dev/qubes_dom0/vm-fedora-33-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-fedora-33-dvm-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-fedora-33-dvm-private'
100.00%

real    0m5.429s
user    0m4.215s
sys 0m0.962s

Restoring wyng-backup archive: vm-default-mgmt-dvm-private into Thin LVM volume: /dev/qubes_dom0/vm-default-mgmt-dvm-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-default-mgmt-dvm-private --save-to /dev/qubes_dom0/vm-default-mgmt-dvm-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-default-mgmt-dvm-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-default-mgmt-dvm-private'
100.00%

real    0m1.603s
user    0m0.467s
sys 0m0.858s

Restoring wyng-backup archive: vm-debian-10-root into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-debian-10-root --save-to /dev/qubes_dom0/vm-debian-10-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-debian-10-root 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-debian-10-root'
100.00%

real    6m1.039s
user    5m35.197s
sys 0m17.920s

Restoring wyng-backup archive: vm-debian-10-private into Thin LVM volume: /dev/qubes_dom0/vm-debian-10-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-debian-10-private --save-to /dev/qubes_dom0/vm-debian-10-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-debian-10-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-debian-10-private'
100.00%

real    0m1.421s
user    0m0.427s
sys 0m0.851s

Restoring wyng-backup archive: vm-whonix-ws-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-ws-15-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-ws-15-root --save-to /dev/qubes_dom0/vm-whonix-ws-15-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-ws-15-root 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-whonix-ws-15-root'
100.00%

real    2m35.438s
user    2m20.510s
sys 0m11.209s

Restoring wyng-backup archive: vm-whonix-gw-15-root into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-gw-15-root --save-to /dev/qubes_dom0/vm-whonix-gw-15-root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-gw-15-root 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-whonix-gw-15-root'
100.00%

real    1m45.752s
user    1m32.826s
sys 0m9.872s

Restoring wyng-backup archive: vm-untrusted-private into Thin LVM volume: /dev/qubes_dom0/vm-untrusted-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-untrusted-private --save-to /dev/qubes_dom0/vm-untrusted-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-untrusted-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-untrusted-private'
100.00%

real    0m1.379s
user    0m0.448s
sys 0m0.807s

Restoring wyng-backup archive: vm-whonix-gw-15-private into Thin LVM volume: /dev/qubes_dom0/vm-whonix-gw-15-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-whonix-gw-15-private --save-to /dev/qubes_dom0/vm-whonix-gw-15-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-whonix-gw-15-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-whonix-gw-15-private'
100.00%

real    0m1.309s
user    0m0.417s
sys 0m0.815s

Restoring wyng-backup archive: vm-windows10_office365proplus-private into Thin LVM volume: /dev/qubes_dom0/vm-windows10_office365proplus-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-windows10_office365proplus-private --save-to /dev/qubes_dom0/vm-windows10_office365proplus-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-windows10_office365proplus-private 20210809-163345
Saving to logical volume '/dev/qubes_dom0/vm-windows10_office365proplus-private'
100.00%

real    0m1.341s
user    0m0.425s
sys 0m0.855s

Restoring wyng-backup archive: root-autosnap into Thin LVM volume: /dev/qubes_dom0/root...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive root-autosnap --save-to /dev/qubes_dom0/root --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : root-autosnap 20210810-114501
Saving to logical volume '/dev/qubes_dom0/root'
100.00%
Diff bytes: 479461376

real    4m7.233s
user    2m43.488s
sys 1m5.824s

Restoring wyng-backup archive: vm-wyng-rsync_net-Insurgo-private into Thin LVM volume: /dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-wyng-rsync_net-Insurgo-private --save-to /dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-wyng-rsync_net-Insurgo-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-wyng-rsync_net-Insurgo-private'
100.00%

real    0m13.492s
user    0m4.674s
sys 0m8.289s

Restoring wyng-backup archive: vm-personal-private into Thin LVM volume: /dev/qubes_dom0/vm-personal-private...
Running command: time sudo ./wyng-backup-fix03/wyng -u --from=internal:/ --subdir=media/wyng-backups/home/user/wyng-backups receive vm-personal-private --save-to /dev/qubes_dom0/vm-personal-private --sparse-write
Wyng 0.3.0wip 20210710
Receiving volume : vm-personal-private 20210810-081324
Saving to logical volume '/dev/qubes_dom0/vm-personal-private'
100.00%

real    0m1.412s
user    0m0.435s
sys 0m0.855s

last commit receive, with wyng-extract-automatized.sh script above and requested output:

[liveuser@localhost-live isoscan]$ ./wyng-extract-automatized.sh 
gpg: assuming signed data in 'backups.sha256sum'
gpg: Signature made Tue 10 Aug 2021 09:26:41 AM EDT
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
Backup signature is VALID

Public key of insugo@riseup.net at the time of this writing was:
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
This should match the public key fingerprint seen above from previous signature verification.
This public key can be downloaded/verified against downloadable links provided under GPG section at https://insurgo.ca#contact

./boot.bzip2: FAILED
./wyng-backups/wyng.backup/default/root-autosnap/volinfo: FAILED
sha256sum: ./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/info: No such file or directory
./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/info: FAILED open or read
sha256sum: ./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/manifest: No such file or directory
./wyng-backups/wyng.backup/default/root-autosnap/S_20210809-163345/manifest: FAILED open or read
sha256sum: WARNING: 2 listed files could not be read
sha256sum: WARNING: 2 computed checksums did NOT match
INTEGRITY OF BACKUP MANIFEST WAS TAMPERED WITH!
Press Enter to continue or CTRL-C to abort...

Restoring wyng-backup archive: vm-windows10_office365proplus-root into Thin LVM volume: /dev/qubes_dom0/vm-windows10_office365proplus-root...
Running command: time sudo ./wyng-extract.sh -s -o /dev/qubes_dom0/vm-windows10_office365proplus-root /media/wyng-backups/home/user/wyng-backups/wyng.backup/ vm-windows10_office365proplus-root
Wyng archive extractor, V0.3.x 20210802
Getting metadata for volume vm-windows10_office365proplus-root.
Volume size = 32212254720 bytes.
  Volume group "dm-9" not found.
  Cannot process volume group dm-9
Hashing volume 31943819264 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    19m23.296s
user    34m18.056s
sys 2m7.316s

Restoring wyng-backup archive: vm-sys-firewall-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-firewall-private...
Running command: time sudo ./wyng-extract.sh -s -o /dev/qubes_dom0/vm-sys-firewall-private /media/wyng-backups/home/user/wyng-backups/wyng.backup/ vm-sys-firewall-private
Wyng archive extractor, V0.3.x 20210802
Getting metadata for volume vm-sys-firewall-private.
Volume size = 2147483648 bytes.
  Volume group "dm-34" not found.
  Cannot process volume group dm-34
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.592s
user    0m14.752s
sys 0m4.274s

Restoring wyng-backup archive: vm-sys-usb-private into Thin LVM volume: /dev/qubes_dom0/vm-sys-usb-private...
Running command: time sudo ./wyng-extract.sh -s -o /dev/qubes_dom0/vm-sys-usb-private /media/wyng-backups/home/user/wyng-backups/wyng.backup/ vm-sys-usb-private
Wyng archive extractor, V0.3.x 20210802
Getting metadata for volume vm-sys-usb-private.
Volume size = 2147483648 bytes.
  Volume group "dm-61" not found.
  Cannot process volume group dm-61
^Cshing volume 0 
bzip2: Control-C or similar caught, quitting.
bzip2: Deleting output file ./00122.bz2, if it exists.
bzip2: WARNING: some files have not been processed:
bzip2:    514 specified on command line, 62 not processed yet.

bzip2: Control-C or similar caught, quitting.
bzip2: Deleting output file ./00136.bz2, if it exists.
bzip2: WARNING: some files have not been processed:
bzip2:    510 specified on command line, 66 not processed yet.

[liveuser@localhost-live isoscan]$ time sudo ./wyng-extract.sh -s -o /dev/qubes_dom0/vm-sys-usb-private /media/wyng-backups/home/user/wyng-backups/wyng.backup/ vm-sys-usb-private
Wyng archive extractor, V0.3.x 20210802
Getting metadata for volume vm-sys-usb-private.
Volume size = 2147483648 bytes.
  Volume group "dm-61" not found.
  Cannot process volume group dm-61
Hashing volume 1879048192 
Creating diff index...
Filling zeros...
Checking volume hashes...sha256sum: 'standard input': no properly formatted SHA256 checksum lines found

real    0m10.683s
user    0m14.696s
sys 0m4.439s
[liveuser@localhost-live isoscan]$ cat /tmp/wyng-extract/diff-manifest
0 x000000007ffc0000 S_8

tlaurion commented 2 years ago

@tasket any update?

tlaurion commented 2 years ago

Again, wrapper around qubes tool would be needed, while having something to deal with /boot backup in the same way would also be required to have a snapshot that is whole in time and be restorable for someone wanting to get back into known qubes state.

tasket commented 2 years ago

@tlaurion

On the error:

I'll probably need some way to emulate/approximate the Heads minimal environment before I can make further progress. Could use some suggestions or help w that (and I realize it may not require a real Heads install, just something that uses the same busybox + set of libraries).

tlaurion commented 2 years ago

@tasket on above errors, it was from Fedora live desktop, so not busybox.

Chosen Tails after, simply because ISO verified from Heads. Same result.

tasket / wyng-backup

Fixes to wyng-extract.sh #87