marmarek
commented
9 years ago Modified by joanna on 14 May 2014 10:38 UTC
Open marmarek opened 9 years ago
marmarek
commented
9 years ago Modified by joanna on 14 May 2014 10:38 UTC
marmarek
commented
9 years ago Comment by joanna on 14 May 2014 10:42 UTC Discussion here:
https://groups.google.com/d/msg/qubes-devel/Gcrb7KQVcMk/CK-saQU_1HYJ
marmarek
commented
8 years ago @Rudd-O in https://github.com/QubesOS/qubes-issues/issues/1588#issuecomment-225452668
Honestly, the backup tool should be replaced by something like Duplicity. They get it right, they do incremental backups, they do encryption, and they do arbitrary targets, so it would be extremely easy to accommodate backing up to another VM, or even to damn S3 if we wanted to.
Ideally, the way I would see that working is:
- Take snapshot of file system containing VMs. Mount the snapshot somewhere, read-only.
- Execute duplicity pointing it to the read-only VM data source, and the target storage destination (VM or local mountpoint)
- Destroy snapshot.
This would also allow for backup of VMs that are running, so no need to shut down VMs during backup. I highly recommend we research replacing qvm-backup with Duplicity.
It looks like Duplicity supports making an incremental backup even when part of a file was changed (include diff of file, not full changed files). So indeed it may be good idea to somehow use it.
But mounting VM filesystem in dom0 is a big NO-GO. On the other hand, it may be good idea to simply run duplicity in the VM, and collect its output. Take a look at the linked discussion for an idea how to handle powered off VMs (in short: launch minimal "stubdomain" like system with access to its disk for the backup purpose).
Of course all this requires evaluation whether duplicity correctly handle encryption / validation. To not make things worse than they currently are...
Rudd-O
commented
8 years ago On 06/12/2016 07:04 PM, Marek Marczykowski-Górecki wrote:
@Rudd-O https://github.com/Rudd-O in #1588 (comment) https://github.com/QubesOS/qubes-issues/issues/1588#issuecomment-225452668
Honestly, the backup tool should be replaced by something like Duplicity. They get it right, they do /incremental/ backups, they do /encryption/, and they do arbitrary targets, so it would be extremely easy to accommodate backing up to another VM, or even to damn S3 if we wanted to. Ideally, the way I would see that working is: 1. Take snapshot of file system containing VMs. Mount the snapshot somewhere, read-only. 2. Execute duplicity pointing it to the read-only VM data source, and the target storage destination (VM or local mountpoint) 3. Destroy snapshot. This would also allow for backup of VMs that are running, so no need to shut down VMs during backup. I highly recommend we research replacing qvm-backup with Duplicity.It looks like Duplicity supports making an incremental backup even when part of a file was changed (include diff of file, not full changed files). So indeed it may be good idea to somehow use it.
But mounting VM filesystem in dom0 is a big NO-GO. On the other hand, it may be good idea to simply run duplicity in the VM, and collect its output. Take a look at the linked discussion for an idea how to handle powered off VMs (in short: launch minimal "stubdomain" like system with access to its disk for the backup purpose).
No need to mount any VM filesystem in dom0. That is also not how Duplicity works either.
The way it would work with Duplicity, is a backend would need to be written. This backend would have two main functions, really:
See this:
https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/files/head:/duplicity/backends/
Later today, or perhaps tomorrow, I will use my Qubes bombshell-client to simulate an SSH connection to the backup VM, and see how Duplicity behaves using this false SSH. That can serve as a good starting point.
Rudd-O
http://rudd-o.com/Better still:
Remarkably much like my Qubes Ansible plugin.
Rudd-O
http://rudd-o.com/Can you explain more how exactly that would work?
private.img file, individual files from within?)Generally dom0 shouldn't be exposed to VM filesystem in any way (regardless of the form: mounting directly, accessing via sftp-like service etc). If incremental data needs to be computed at VM-file level, it should be done in separate VM and dom0 should treat the result as opaque blob. Also during restore.
Rudd-O
commented
8 years ago On 06/12/2016 08:15 PM, Marek Marczykowski-Górecki wrote:
Can you explain more how exactly that would work?
- Where and at what level VM data would be retrieved (|private.img| file, individual files from within?)
Duplicity + hypotheticalplugin (from now on, HP) would never retrieve any VM data. It would only retrieve an encrypted file from the backup directory within the VM. This encrypted file contains a manifest of what exists as a backup, as well as a set of rdifflib diff informations.
- Where and at what level incremental data is computed?
Incremental data is computed in dom0 using both the rdifflib database and the actual contents of the snapshotted VM images. This should be safe because the database is stored encrypted, so the VM cannot tamper with it and induce a code execution in dom0.
Generally dom0 shouldn't be exposed to VM filesystem in any way (regardless of the form: mounting directly, accessing via sftp-like service etc). /If/ incremental data needs to be computed at VM-file level, it should be done in separate VM and dom0 should treat the result as opaque blob. Also during restore.
That cannot work because that involves copying the IMG files into the separate VM, and that would take forever.
Duplicity's backends are designed to make the storage opaque to Duplicity. The backend retrieves files and uploads files, and those files are encrypted by the time the data reaches the backend plugin. The backend does not care about anything else.
Rudd-O
http://rudd-o.com/That cannot work because that involves copying the IMG files into the separate VM, and that would take forever.
Not necessary - it can be attached with qvm-block-like mechanism.
Anyway, IIUC you want to backup *.img files in dom0 using duplicity. This may work. My initial (maybe invalid) concern about input validation on restore is still stands.
Rudd-O
commented
8 years ago Well, look at what I have got here:
(deleted cos buggy, see below for updated version)I have a (shit) complete Qubes VM backend for duplicity which you can add to your dom0's duplicity backends directory, and then run something like this:
duplicity /var/lib/qubes qubesvm://backupvm/Backupsmountpoint/mydom0/qubesvms
Very early working prototype and building block. Enjoy!
v6ak
commented
8 years ago I have some experience with Duplicity:
I would prefer using Duplicity per-VM, so one could exclude caches etc. from the backup. My idea is something like the following process invoked from dom0:
private.img to the BackupDVM.As you can see, this would require some data stored in dom0 to be also backed up, but this could be handled by some very similar mechanism. (It would probably run directly in dom0 rather than in dom0.)
Some implementation and security notes on those points:
/usr/lib/qubes/qfile-daemon-dvm.\A[a-z][a-z0-9.-]*\Z would be enough for Duplicity) and send it to some actual Duplicity backup storage backend (e.g. rsync, SFTP, Amazon S3, Google Drive, …).Rudd-O
commented
8 years ago --gpg-options '--cipher-algo=AES256 --digest-algo=SHA256' to fix that.duplicity does talk about it.qubesvm:// see below) and Duplicity will back up the source dir to the dest dir. Presumably we would add value by creating a system of "profiles" (each defined as "what to back up, and where to") where the user can use a GUI or a config file to establish a routine backup plan (perhaps automatic) of one or more of these profiles./usr/lib64/python2.7/site-packages/duplicity/backends/qubesvmbackend.py. Then your destination URL in your duplicity command line can be qubesvm://yourbackupvm/path/to/directory/to/store/backup. I backed up /var/lib/qubes (technically, a mounted snapshot of it) with no problem to a qubesvm://mybackupvm/mnt/disk/qubesbackup directory.# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# This file is NOT part of duplicity. It is an extension to Duplicity
# that allows the Qubes OS dom0 to back up to a Qubes OS VM.
#
# Duplicity is free software, and so is this file. This file is
# under the same license as the one distributed by Duplicity.
import os
import pipes
import subprocess
import duplicity.backend
from duplicity.errors import *
BLOCKSIZE = 1048576 # for doing file transfers by chunk
MAX_LIST_SIZE = 10485760 # limited to 10 MB directory listings to avoid problems
class QubesVMBackend(duplicity.backend.Backend):
"""This backend accesses files stored on a Qubes OS VM. It is intended to
work as a backed within a Qubes OS dom0 (TCB) for the purposes of using
Duplicity to back up to a VM. No special tools are required other than
this backend file itself installed to your Duplicity backends directory.
Missing directories on the remote (VM) side will NOT be created. It is
an error to try and back up to a VM when the target directory does
not already exist.
module URL: qubesvm://vmname/path/to/backup/directory
"""
def __init__(self, parsed_url):
duplicity.backend._ensure_urlparser_initialized()
duplicity.backend.urlparser.uses_netloc.append("qubesvm")
duplicity.backend.urlparser.clear_cache()
properly_parsed_url = duplicity.backend.ParsedUrl(parsed_url.geturl())
duplicity.backend.Backend.__init__(self, properly_parsed_url)
if properly_parsed_url.path:
self.remote_dir = properly_parsed_url.path
else:
self.remote_dir = '.'
self.hostname = properly_parsed_url.hostname
def _validate_remote_filename(self, op, remote_filename):
if os.path.sep in remote_filename or "\0" in remote_filename:
raise BackendException(
("Qubes VM %s failed: path separators "
"or nulls in destination file name %s") % (
op, remote_filename))
def _dd(self, iff=None, off=None):
cmd = ["dd", "status=none", "bs=%s" % BLOCKSIZE]
if iff:
cmd.append("if=%s" % iff)
if off:
cmd.append("of=%s" % off)
return cmd
def _execute_qvmrun(self, cmd, stdin, stdout):
subcmd = " ".join(pipes.quote(s) for s in cmd)
cmd = ["qvm-run", "--pass-io", "--", self.hostname, subcmd]
return subprocess.Popen(
cmd,
stdin=stdin,
stdout=stdout,
bufsize=MAX_LIST_SIZE,
close_fds=True
)
def put(self, source_path, remote_filename=None):
"""Transfers a single file to the remote side."""
if not remote_filename:
remote_filename = source_path.get_filename()
self._validate_remote_filename("put", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(off=rempath)
fobject = open(source_path.name, "rb")
try:
p = self._execute_qvmrun(cmd,
stdin=fobject,
stdout=open(os.devnull))
except Exception, e:
raise BackendException(
"Qubes VM put of %s (as %s) failed: (%s) %s" % (
source_path.name, remote_filename, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM put of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
source_path.name, remote_filename, err))
def get(self, remote_filename, local_path):
"""Retrieves a single file from the remote side."""
self._validate_remote_filename("get", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(iff=rempath)
fobject = open(local_path.name, "wb")
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull),
stdout=fobject)
except Exception, e:
raise BackendException(
"Qubes VM get of %s (as %s) failed: (%s) %s" % (
remote_filename.name, local_path, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM get of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
remote_filename.name, local_path, err))
def _list(self):
"""Lists the contents of the one duplicity dir on the remote side."""
cmd = ["find", self.remote_dir, "-maxdepth", "1", "-print0"]
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=subprocess.PIPE)
except Exception, e:
raise BackendException(
"Qubes VM list of %s failed: %s" % (self.remote_dir, e))
data = p.stdout.read(MAX_LIST_SIZE)
p.stdout.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM list of %s failed: list command finished "
"with nonzero status %s" % (self.remote_dir, err)))
if not data:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"empty" % (self.remote_dir,)))
filename_list = data.split("\0")
if filename_list[0] != self.remote_dir:
raise BackendException(
("Qubes VM list of %s failed: list command returned a "
"filename_list for a path different from the remote folder") % (
self.remote_dir,))
filename_list.pop(0)
if filename_list[-1]:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"wrongly-terminated data or listing was too long") % (
self.remote_dir,))
filename_list.pop()
filename_list = [ p[len(self.remote_dir) + 1:] for p in filename_list ]
if any(os.path.sep in p for p in filename_list):
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"a path separator in the listing") % (
self.remote_dir,))
return filename_list
def delete(self, filename_list):
"""Deletes all files in the list on the remote side."""
if any(os.path.sep in p or "\0" in p for p in filename_list):
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command asked to delete a file with a path separator "
"or a null character in the listing") % (
self.remote_dir,))
pathlist = [os.path.join(self.remote_dir, p) for p in filename_list]
cmd = ["rm", "-f", "--"] + pathlist
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=open(os.devnull, "wb"))
except Exception, e:
raise BackendException(
"Qubes VM delete of files in %s failed: %s" % (
self.remote_dir, e))
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command finished with nonzero status %s") % (
self.remote_dir, err))
duplicity.backend.register_backend("qubesvm", QubesVMBackend)Finally: I disagree with the direction of implementation which suggests we need to play Towers of Hanoi with a backup VM and attaching disk images to it. That's entirely unnecessary complication, and it also demands VMs be off for backup purposes. Entirely unnecessary and extremely discouraging of backup procedures.
The three step process is all that is necessary:
/var/lib/qubes, then mount somewhere./mnt/snapshot/var/lib/qubes (I believe there's even an option to trim the mountpoint out of the sent paths, much like tar --strip-components).That is all that is needed.
Of course, the Qubes setting of which VMs to back up, plus options like -x that are supported in qvm-backup, would also be sensible things to support in the wrapper W.
Rudd-O
commented
8 years ago Update: Duplicity is no good for the process, because differential backups slow to a crawl.
We need to research other stuff, like Attic.
v6ak
commented
8 years ago I have not experienced such issue with Duplicity. I have managed to perform incremental backup on roughly 15GiB (after exclusions) of various data (many small files and few large files) in ~2 minutes even on HDD + dm-crypt. (Of course, this depends on size of changes.) Encryption and compression were enabled.
Maybe it skips many files just because of timestamps and diffs just few files.
So, I feel you must be doing something wrong. (No offense.)
What was your scenario? What files did you try to backup? (E.g., dom0's ~ with 1GiB of small files.) What was your drive setup? (E.g., 7200 RPM HDD with dm-crypt with ext4.) How much time did it take? Where do you store the backup? (Other VM using your storage backend? Well, theoretically, this should not matter so much for performance of scanning, as Duplicity caches metadata locally, AFAIK somewhere in ~/.cache. But I am curious.) Did you have the metadata cached? How much data did you add to the backup? (Well, if you backup ~ and you don't exclude ~/.cache, you might add Duplicity metadata to the backup, which could explain both some time and space penalty. I am not sure if Duplicity is smart enough to exclude this automagically.)
Rudd-O
commented
8 years ago On 07/19/2016 05:45 PM, Vít Šesták wrote:
I have not experienced such issue with Duplicity. I have managed to perform incremental backup on roughly 15GiB (after exclusions) of various data (many small files and few large files) in ~2 minutes even on HDD + dm-crypt. (Of course, this depends on size of changes.) Encryption and compression were enabled.
My backup of a 60 GB disk image progressed essentially nothing over the course of five hours. I think there is an exponential slowdown after a certain size.
Rudd-O
http://rudd-o.com/Inspired by lecture pointed by @Rudd-O in parallel ticket, what about this:
That would be:
easy to recover the data even without Qubes, something like this should do:
tar xSOf private.img.tar.full > private.img
tar xSOf private.img.tar.incremental | dd of=private.img conv=sparse,notruncSome details to work out:
Any thoughts? @Rudd-O @defuse @jpouellet
jpouellet
commented
7 years ago @marmarek I like the simplicity and elegance, but unfortunately there is a problem with that.
Blocks which have been zeroed, backed up, then restored, will not be faithfully reproduced. In the source of the incremental backup the block contains zeros, but in the restored image it would contain whatever it contained before being zeroed.
Consider this demonstration:
$ perl -e 'print "A"x(1024*4)' > A
$ perl -e 'print "B"x(1024*4)' > B
$ perl -e 'print "C"x(1024*4)' > C
$ perl -e 'print "\x00"x(1024*4)' > 0
$ hexdump -C A
00000000 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
*
00001000
$ hexdump -C B
00000000 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 |BBBBBBBBBBBBBBBB|
*
00001000
$ hexdump -C C
00000000 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 |CCCCCCCCCCCCCCCC|
*
00001000
$ hexdump -C 0
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000
$ cat A 0 C > full
$ cat A B 0 > incremental
$ dd if=full of=restore
$ cat full > restore
$ cat incremental | dd of=restore conv=sparse,notrunc
24+0 records in
24+0 records out
12288 bytes (12 kB) copied, 8.6333e-05 s, 142 MB/s
$ hexdump -C full
00000000 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
*
00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 |CCCCCCCCCCCCCCCC|
*
00003000
$ hexdump -C incremental
00000000 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
*
00001000 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 |BBBBBBBBBBBBBBBB|
*
00002000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00003000
$ hexdump -C restore
00000000 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|
*
00001000 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 |BBBBBBBBBBBBBBBB|
*
00002000 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 |CCCCCCCCCCCCCCCC|
*
00003000In other words, if "unchanged" is encoded as "skip this" which is decoded as zeroes, and then all zeroes are interpreted as "skip this", then you have an unintended consequence of zeroed data in your source image being interpreted as "unchanged" rather than "been changed to zeroes". We would need a trustworthy way of disambiguating the two.
marmarek
commented
7 years ago @jpouellet I see, but the problem is only only with simplified restore path - because dd cannot distinguish between zeroed block and block not included in the archive. But the information is there. In fact the only function of dd here is to avoid truncating when restoring to a file. It isn't needed when restoring to block device. So - probably better would be to use loop device instead of dd. Or find some tar option which I'm not aware of.
Mmm, maybe this trick would work:
tar xSOf private.img.tar.full > private.img
exec 3<>private.img
tar xSOf private.img.tar.incremental >&3On 11/04/2016 10:04 AM, Marek Marczykowski-Górecki wrote:
Any thoughts? @Rudd-O https://github.com/Rudd-O @defuse https://github.com/defuse @jpouellet https://github.com/jpouellet
Your scheme sounds correct.
Unfortunately, I detest full+incremental backup series — I prefer an opaque encrypted block store that can be added to, and cleaned up, because it's more efficient and I do not need to diddle / fuck around with tracking full+incremental series. It's why I was looking at Attic, until I found out about Duplicati.
The implementation and algorithm suggested by Duplicati — with a minor tweak to add privacy by preventing deduplication between VMs — seems much better, more efficient, faster, and cryptographically sound.
I understand that, as the software is today, it does not fulfill the requirements of Qubes (restore with no extra tools), and I respect that.
Rudd-O
http://rudd-o.com/On 11/05/2016 12:33 AM, Marek Marczykowski-Górecki wrote:
@jpouellet https://github.com/jpouellet I see, but the problem is only only with simplified restore path - because |dd| cannot distinguish between zeroed block and block not included in the archive. But the information is there. In fact the only function of |dd| here is to avoid truncating when restoring to /a file/. It isn't needed when restoring to block device. So - probably better would be to use loop device instead of |dd|. Of find some |tar| option which I'm not aware of. Mmm, maybe this trick would work:
|tar xSOf private.img.tar.full > private.img exec 3<>private.img tar xSOf private.img.tar.incremental >&3 |
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-258578577, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVIwr5gUlMuJfc4I4MPElrW2yfll__Aks5q687IgaJpZM4DrUhq.
Honestly, it's become time to write a proper restore tool. The tool I wrote and posted on the other bug does precisely this — skips over the blocks that are zeroes. Unfortunately, it looks like that tool won't work for this purpose because you may need byte-level granularity?
Rudd-O
http://rudd-o.com/
tasket
commented
7 years ago The approach used by Apple's Time Machine has great merit, enjoying all the benefits being sought here:
It even allows mounting of encrypted volumes remotely.
Apple's approach is valuable to Qubes because they catered to their users' tendency to create very large files (usually media projects) often with small, frequent changes which rendered traditional incremental backup tools inefficient.
To get TM working with encryption, Apple had the user's home folder mounted from a 'sparsebundle'... a disk image that is actually a folder with a bunch of encrypted 2MB files. When a virtual disk block within any given 2MB 'band' is written, the band's backing file in the sparsebundle's folder would naturally have its mtime updated. --- That is the only 'special sauce' needed to implement backups that work like Time Machine.
The system only has to remember the time of the last backup and scan the sparsebundle dir by mtime to quickly find the changed data. For completeness, each session folder on the destination gets the unmodified bands from the prior backup in the form of hardlinks; This makes each backup increment whole and yet simple to manage (deleting a whole backup session won't delete a data band unless that folder was the only one referencing it).
The trick for Qubes would be to have the guest VM storage layer emulate the sparsebundle, with its band files--though examples do exist on Linux. If actual image files aren't involved, use a block driver capable of storing a bitmap to flag each virtual band as its modified.
Nearly all the other techniques involve processing the entirety of the source data during each backup. That is too CPU-, disk- and time-intensive to saddle on laptop users.
One decent alternative would be to utilize the more advanced Linux storage layers: Thin provisioned LVM and Btrfs can both quickly discern deltas between snapshots, resulting in efficient backup processing. Btrfs comes with btrfs-send which handles this function, and LVM has example code available. The downsides are that old backups cannot be sent in encrypted form and integrated/pruned on the destination unless the VM data is already encrypted as separate LUKS/dmcrypt containers; the live system also has to use extra space to hold the prior-backup's snapshot.
marmarek
commented
7 years ago This is interesting idea, but not directly applicable to Qubes, without (IMHO too large) compromises on privacy and complexity.
Storing VM disk image as 2MB files (or such) seems technically hard - I don't know any sane Linux method to do that. Insane one: create loop device for each part and combine using device mapper. Or: check if qemu could do that (we don't want qemu in dom0).
This method implies more trust in backup storing facility. Mostly about privacy (backup observer can see what part of data have you changed and make some conclusions based on this), but also handling integrity would be more complex (need to assure that parts archived during very different backup runs are not manipulated, reordered etc). This is the price for easier cleanup of old backups. IMHO too high.
But indeed combining some smart method of listing modified blocks (instead of comparing hashes) with https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-258388616 would be nice. Can you point exact info how to get modified block bitmap between two LVM thin snapshots or btrfs equivalent?
Additional think to learn from this - Apple have chosen 2MB chunks, I guess this is based on some research, so I'd reuse it.
tasket
commented
7 years ago @marmarek : Maybe 'insane' on Linux but not BSD... for some reason? ...don't know... :)
Here is Apple's open source code that makes it possible, in 480 lines:
http://www.opensource.apple.com/source/hfs/hfs-195/CopyHFSMeta/SparseBundle.c
That's it. Once you have that, you can do backups of VM image deltas just like 'rsnapshot' and several others would normally do with individual files on Linux. The destination just has to support hardlinking, and plenty do nowadays. Performance wise, it has been acceptable for millions of Mac users.
https://github.com/jeffmahoney/sparsebundle-loopback https://github.com/torarnv/sparsebundlefs https://github.com/vasi/rhfs
I am also not against what you're saying on this point, and I think a "minimally incremental" process that needs to re-do a full backup periodically may ultimately be preferable. But there is a more manageable way to do it that would allow people to backup hourly without pain.
Also, the implementation does not have to be with file bundles... It can be 99.9% normal Linux block device using LVM or single img file, but with some extra handling of a bitmap. So instead of mtimes in a file bundle, you are checking for 'changed' bits in the bitmap.
If you don't go with the Time Machine approach, you should at least consider using LVM/Btrfs snapshots as the means of finding volume deltas. If I know my system needs to backup 200MB of changes, I do not want to see it wheezing at full throttle for 40 minutes to get there... I'd rather just have a slightly larger disk that can handle the snapshots and backup in less than 5 minutes.
tasket
commented
7 years ago @marmarek : Also, before going too much out on a limb... IIRC in the old bugtrack ticket Joanna said she thought the TM-style approach sounded good. That's how I remembered it.
marmarek
commented
7 years ago I thought of the dm idea, but it isn't made for that. As Apple did, a block device is necessary... and not hard to do. Here are other implementations:
https://github.com/jeffmahoney/sparsebundle-loopback https://github.com/torarnv/sparsebundlefs https://github.com/vasi/rhfs
I wonder about performance of those. In most cases FUSE isn't good at it. We already have I/O performance reduced enough...
I am also not against what you're saying on this point, and I think a "minimally incremental" process that needs to re-do a full backup periodically may ultimately be preferable. But there is a more manageable way to do it that would allow people to backup hourly without pain.
Yes, this.
If you don't go with the Time Machine approach, you should at least consider using LVM/Btrfs snapshots as the means of finding volume deltas. If I know my system needs to backup 200MB of changes, I do not want to see it wheezing at full throttle for 40 minutes to get there... I'd rather just have a slightly larger disk that can handle the snapshots and backup in less than 5 minutes.
Yes, I'd like to (as we'll have snapshots in place in Qubes 4.0 anyway). This is why I've asked how to extract LVM/Btrfs volume delta (modified blocks bitmap or such).
tasket
commented
7 years ago Yes, I'd like to (as we'll have snapshots in place in Qubes 4.0 anyway). This is why I've asked how to extract LVM/Btrfs volume delta (modified blocks bitmap or such).
Have you seen this yet? https://github.com/jthornber/thinp-test-suite/blob/master/incremental_backup_example
For Btrfs, you can look at the btrfs-send source. You could also just use btrfs-send itself, as it outputs a stream.
tasket
commented
7 years ago Re: Block devices, it doesn't have to be FUSE. It can be done as a kernel driver, or even as a modification to the existing driver.
marmarek
commented
7 years ago thin_delta seems to be exactly what is needed. A bit low level, but well...
Note to self - required operations:
# Note snapshot device ids:
lvdisplay -m qubes_dom0/VMNAME-private-snapshot1
lvdisplay -m qubes_dom0/VMNAME-private-snapshot2
# create "pool metadata snapshot" - required by thin_delta tool to access active pool
dmsetup message /dev/mapper/qubes_dom0/pool0-tpool 0 reserve_metadata_snap
# Compare snapshots - output is XML
thin_delta --thin1 <first snapshot id> --thin2 <second snapshot id> -m /dev/mapper/qubes_dom0-pool0_tmeta
# release "pool metadata snapshot"
dmsetup message /dev/mapper/qubes_dom0/pool0-tpool 0 release_metadata_snapThe downside of having full snapshot vs just block hashes is of course higher disk usage. Some may be willing to pay this price, some not - I guess it will depends on actual additional space needed and full backup frequency for particular user.
Rudd-O
commented
7 years ago On 11/22/2016 10:30 AM, tasket wrote:
The approach used by Apple's Time Machine has great merit, enjoying all the benefits being sought here:
- Quick handling of very large files
YES!
Nearly all the other techniques involve /processing the entirety of the source data during each backup/. That is too CPU-, disk- and time-intensive to saddle on laptop users.
Indeed. This is why incremental backups using duplicity (the code of which I wrote) take FOREVER.
I would prefer something like a btrfs delta snapshot being backed up, because that's even faster than what Mac OS X does. But it doesn't seem to be the case that anyone at ITL wants to implement that.
Rudd-O
http://rudd-o.com/I am working on a PoC of incremental backups. Currently, it just handles key derivation, cloning the imgs, launching of the DVMs, and encrypting the VM name, but I believe that the design it is clear ATM.
andrewdavidwong
commented
7 years ago @v6ak's PoC with request for feedback: https://groups.google.com/d/msgid/qubes-users/901b82dc-f781-4c13-ad00-33b4337fc84a%40googlegroups.com
enkore
commented
7 years ago Update: Duplicity is no good for the process, because differential backups slow to a crawl.
We need to research other stuff, like Attic.
(Just btw./FYI) the maintained fork of Attic is Borg
Works generally fine with VM images (as opposed to working directly inside the VM filesystem), but the disadvantage that you can't selectively exclude stuff from an image applies as well, and that extra hops are needed to access files inside an image (at least a loop device).
I've had the idea that one could run borg in dom0 to create the backups but use BORG_RSH="qrexec..." to access a repository (local/remote) via any AppVM.
For the next major version (1.1) we created some docs about crypto in Borg and other aspects to security: http://borgbackup.readthedocs.io/en/latest/internals/security.html
DemiMarie
commented
6 years ago Are there any plans to implement this? This is very important for use of Qubes in environments that do not have very fast Internet connections. I have not backed up my system at all yet, for instance.
This seems like a simple task: take LVM snapshots in dom0, diff them for each backup, then encrypt the diffs and send them.
mirrorway
commented
6 years ago As mentioned above, you can use borgbackup.
More details here: https://groups.google.com/d/msg/qubes-users/cNC-dthpB1o/pFVTXHfXAgAJ
tasket
commented
5 years ago I have the beginnings of a proof of concept for using dom0 volume metadata available from LVM thin_delta to track volume changes in block segments. This means no exposure to domU malware, no grinding through hash based full-data comparisons, no long term space-eating snapshots sitting on the computer (snapshots used are short-term and regularly deallocate old data), no being tied to an original/old backup set (infinite pruning). And processing the volume deltas happens very quickly.
Technically, its like a cross between OSX Time Machine and Btrfs Send. It rotates a couple of *private volume thin snapshots on an N-minutes or hourly basis, and uses thin_delta to gather block change (delta) information during each snapshot rotation (this should also be possible with Btrfs tools). The change info is accumulated in an efficient bitmap file which uses only kilobytes even when mapping very large volumes. When backing up, segment files are created for only the changed areas indicated in the bitmap and they are stored on the destination in a directory structure organized by session time, volume, etc.
The main obstacle I have is the way Qubes lumps .cache data in with regular persistent data -- traditional OSes can easily separate them, but Qubes currently binds them together. This is because cache is stored in *private volumes. The churn this creates in the delta map reduces the efficiency gains considerably (the effect on regular qvm-backup and other tools is also rather bad). I opened #4217 to address this issue.
ThomasWaldmann
commented
5 years ago @tasket thanks for your PoC code, I'll soon have a look whether I can assimilate a snapshot-diff based method for borgbackup (or for a wrapper around it). I'm also (finally! \o/) using qubes os since few weeks, needing backups....
tasket
commented
5 years ago @ThomasWaldmann If assuming I were to use sparsebak as a wrapper, the required capability from borg would be to accept a set of named objects (source chunks) as a stream (instead of files on disk). An ideal way to do this may be to accept tar format on stdin, since that's how I'm using tar now, as a streaming transport. Preferably, for restoring it should also have some way to recognize each incremental backup set within a larger namespace representing a whole volume or set of volumes (though storing sparsebak's manifests might be sufficient).
The sparsebak project itself -- soon to be renamed -- is moving past PoC into a beta and eventually final release. It will be a single-file tool for archiving and updating logical volumes. Of course, you're welcome to try it while experimenting with borg refits.
v6ak
commented
5 years ago On Borg: AFAIR, I had some concerns about its encryption scheme. Unless some care is taken, it seemed plausible that it can reuse nonce after some recovery scenarios. The author considers it as a feature, not a bug.
ThomasWaldmann
commented
5 years ago @tasket I need to think more about the problem and the implementation of the solution.
My first idea rather was that borg gets a list of block numbers (or block ranges or offset+length infos) of changed blocks and updates an identified previous state with these changed blocks (reading the blocks by itself). The current content-defined chunker would get an additional alternative implementation of a fixed-blocksize chunker for that.
borg's files cache stores (per file): timestamps, inode, size, list of chunk-hash-IDs. The latter is currently used to quickly process unchanged files without having to read/chunk/hash them again (a borg archive is always logically a full backup). With a fixed block size chunker and a list of changed chunks, the cached list could be updated with the hash-IDs of the changed chunks to create the new full list. The hashid is not a simple hash for encrypted repos, but a MAC computed by borg using the MAC secret key. So we need that memorized list from the files cache.
@v6ak if you want to discuss borg aes ctr trust issues, I suggest you locate or open a separate ticket.
Rudd-O
commented
5 years ago -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
@tasket
If you use ZFS diff or btrfs diff (on dom0), and these gave you actual blocklists of differences, the churn you'd see from LVM thin_delta would likely be largely reduced, because snapshot diffs would not contain deleted cached data. Whether you go with LVM or a WAFL file system, another solution to explore would probably involve injecting into VMs a third block device (100% volatile) to mount over ~/.cache, and the backup tool explicitly excluding that volatile block device. -----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEUUn9BY4ZnowjgNibWZnBtXu9hDYFAlwzTuAACgkQWZnBtXu9 hDYAqg/6ArP3iGeQBzwgzIk2kD14gc3k549NGci0S3M3KwbO6eoeqXHTxkk1VqGp 586yhLlma6gEobYj00JEzZt7Di4Ot0mEs/uOrIhPyU67uubB677EgwB2jC3jYpAS CVrwzq9USigLeceaZXCkS7xOEqzTUogPtSI8DloyBr0kFWftUa6Wp119yYb4T6OM 1+/Ax3LW6gmbgl2CGmzVhi9dgOtn/Ut2Nzpao19hk/jB/56837e2fwT+Y/dKd8kV gfD9LGCInJWYvjsLH0EqOmJF+GWqD/bfmwku3N+GSFwEfW/zWD8SMG3os71Ztan0 uZ5JK+NdAZbagQjtUo/TOuKEpFypJvrNGsfF23+gObR1qVFZfXso+tybJCcnNqVX LCwzuLk+tP1Ws43EmwEpv01idiyge57Eb+X66MmGyPyCGEJOKWWhvhV4AdGc3XMJ ImIkqr/2+CuBfr8mfEYYLMECQlK1EpNd/pCC0W2EIPF1QfF4HbEyPt+BoRbqJDi6 XdAb3hT68b1wzd5RGOB7YAKSBhrvrzY13iCdQNjoiMx/+Ufvabz9jxQ+CAKBvC4I y7rMKHStS833eoxMK3pQeHqs0S0UjALjWYeyRR5hEz/DunURNAcDjGvatlcK+tl9 UBkQgcF8f1zrQbYVsdI6paTJHkDhqZLhdmg+POwuHvlR6MVXGko= =coPk -----END PGP SIGNATURE-----
tasket
commented
5 years ago @Rudd-O Unfortunately, a COW filesystem diff doesn't seem to add to what LVM thin_delta provides -- it discerns between changed and discarded chunks in the form of 'right-only' and 'left-only' tags. So its already ignoring anything that was both created+deleted between adjacent backups.
Additional block device is covered in issue #4217. Marek's suggestion to let users switch-on the additional cache device on a per vm basis sounds OK.
tlaurion
commented
1 year ago Houla.
Sparsebak is now called wyng-backup, is alive and solid in its 0.3 serie.
0.4 serie implements encryption and would need some more users.
@tasket, maybe you can give an update here on the status of the project?
tasket
commented
1 year ago Yes, Wyng v0.3 has been released. This version is "bring your own encryption".
Wyng v0.4 is in development with a usable alpha available. It offers encryption, although it has not yet been reviewed by a cryptographer. Apart from crypto, its intended to be more flexible, more efficient and easier to use. For example, archives are now specified on the command line as a simple URL with the option of using a nickname for the location, and overall metadata footprint is reduced.
Two optional libraries available in Qubes 4.1 dom0 repo are 'python3-pycryptodome' which enables encryption, and 'python3-zstd' which enables zstandard compression. Both libraries are also available in popular Linux distros' default repositories.
Full Qubes integration presents problems because Qubes has no "import VM settings verbatim" function that a restoration process can use, and I consider the qubes-backup-restore method to violate usability and naming (e.g. security) expectations of most users. The first wrapper I made for this purpose was sort of rejected by Marek, so I put it on a back burner. However, I'll be attempting a more pragmatic approach that relies on system default settings.
Wyng can be used without the VM setting integrations, of course, handling VM data only. But it means restoring VM settings has to be done manually. This is how I use it.
Other Qubes integration such as respecting dom0 isolation and using helper domUs for accessing archives is working fine.
BTW, since avoiding .cache data was mentioned, I'll note issue #4217 was created for this purpose.
tlaurion
commented
1 year ago @marmarek (or @andrewdavidwong ) : maybe changing sparsebak to wyng for both urls and name of project would be usefull to get people interested in the project progression.
Maybe also edit OP to point to status update at https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-1217463303
andrewdavidwong
commented
1 year ago @marmarek (or @andrewdavidwong ) : maybe changing sparsebak to wyng for both urls and name of project would be usefull to get people interested in the project progression.
Maybe also edit OP to point to status update at #858 (comment)
Done, thanks!
emanruse
commented
1 year ago Please consider the possibility of integrating Bacula in Qubes OS. I have been using it for many years and I can say it is a wonderful backup system - extremely powerful, configurable, supports tape backup, etc.
tlaurion
commented
1 year ago @emanruse what is Bacula? It backups files or block devices? Is it differential? How does it work? Why would it fit Qubes in dom0?
emanruse
commented
1 year ago what is Bacula?
https://www.bacula.org/what-is-bacula/
It backups files or block devices?
From the manual:
"The Storage daemon reads and writes in units of blocks. Blocks contain records. Each block has a block header followed by records, and each record has a record header followed by record data."
Not sure if that answers your question.
Is it differential?
It can do full, differential and incremental backups. The enterprise version also has a delta compression plugin:
"The Delta plugin is designed to analyze file differences at the block level and generate binary patches for the backup. It saves significant amounts of space on files that have few modifications between two backups." How does it work? Why would it fit Qubes in dom0?
How does it work?
It consists of a Director, File Daemon(s) and Storage Daemon(s) which can work on one or separate (virtual or physical) machines. The Director manages the work of other deamons. You need to read the docs for more.
Why would it fit Qubes in dom0?
That needs research. I still don't know if it is a good fit for Qubes OS because of its (Qubes OS's) specifics. Recently I opened a tread in the forum to discuss that but it doesn't seem to attract much interest. I suppose that's because Qubes OS people don't know about Bacula (which is strange, as it is a really big name in backup software).
My experience with Bacula is with Linux systems (including Windows clients) in a LAN and it does a wonderful job there.
tlaurion
commented
1 year ago @emanruse : thanks for your reply.
The reason why I ask was for people to read here why it would be of a greater interest then https://github.com/tasket/wyng-backup actual candidate (to be integrated into Qubes backup/restore tool), which is already made to be differential backup system, operating at the block (LVM/thin-LVM) level, and permits to store into a seperate dom0 disk/a qube/a ssh server(not Qubes related)/qubes+ssh(having a qube talk and deal with authentication so dom0 uses that qube to interact with ssh from there), encryption being cared about on your own (version 0.3) or with encryption support (0.4), with even zstd support (Facebook lossless codec with best compression ratio as of today). It is blazing fast, does the job pretty well and fits Qubes mindset, today.
I use it for more than a year, did a talk about it https://www.youtube.com/watch?v=1u9Q6ANU51E and is a pretty solid all in one python script (with additioanl python dependencies for zstd and encryption).
The idea here would be to know what Bacula would serve better then wyng, how it would integrate easily under Qubes (without too much added complexity under dom0) to respect Qubes compartmentalization model.
As you can see, this issue was opened in 2015. I personally try to push Qubes into integrating something better than the actual qubes backupo system (which does a great job at backuping everything but dom0, which is another subject) but it would be nice if we could collaborate into improving the amazing work already done to fill the gap instead of bringing additional tools in the debate unless they really do the job better, and actually ease Qubes integration of such tools into the actual Qubes backup/restore tool.
So to ask things differently: have you tried wyng to compare it against your knowledge of Bacula?
Community Devs: @v6ak, @tasket @v6ak's PoC: https://github.com/v6ak/qubes-incremental-backup-poc @tasket's PoC: https://github.com/tasket/wyng-backup | Status update as of 2022-08-16: https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-1217463303
Reported by joanna on 14 May 2014 10:38 UTC None
Migrated-From: https://wiki.qubes-os.org/ticket/858
Note to any contributors who wish to work on this issue: Please either ask for details or propose a design before starting serious work on this.