Move .cache to a separate volume for efficient backups

tasket commented 6 years ago

Qubes OS version:

R4.0

Affected component(s):

domU storage any dom0-based backup tools like qvm-backup

General notes:

A large but seldom-discussed problem we've had with backups is that application cache contents are backed-up along with the normally saved/wanted data in guest VMs. By definition cache is a type of data is only retained between sessions for efficiency, yet it generates a lot of churn, bandwidth and space usage in backup data sets. This is a significant burden on user time and resources.

And with some use cases generating cache churn reaching into the gigabyte(s) range for individual VMs, this problem largely negates the development of efficient incremental backups.

Proposal

Although traditional backup tools commonly avoid cache contents, unfortunately the current practice of storing application cache in *private means this data cannot be avoided by a dom0-based backup tool. A bit of added structure in the virtual storage layer is required in the form of a cache volume.

Cache volume characteristics

Administered in identical fashion to *private volumes for VM create, start, stop, clone, etc.
Mounted at VM startup, with .cache in /rw/home linking or binding to it
Cache volumes are only "volatile" to certain tools, e.g. backups
Cache volumes can also be deleted in system emergencies, such as pool out-of-space situations
May also serve as an info security perk

Treating /home/*/.cache as a separate class of data in this way with its own separate virtual volume would dispel a shadow hanging over Qubes backups: It would immediately improve qvm-backup and enable developments in efficient incremental backups, for example ones based on dom0 COW metadata.

In terms of implementation difficulty, I think this rates as low-moderate. The main difficulty will be in making sure private and cache are snapshotted atomically together.

Related issues:

Issue #858

t4777sd commented 6 years ago

I think it is a good idea to have a bind-dirs type volume that includes non-essential data. If it is done I think it should be similar to the /rw directory and bindable to various directories depending on need. That way there is flexibility. For example, there is a LOT more than just cache data that creates churn. And, not all applications or operating systems are freedesktop compliant so there is no ".cache" folder.

An example of a different class of churn creation would be torrents. People download a movie, watch it, and delete it. Unfortunately it just got backed up. So, the torrent folder should also live on the volume meant for no backups.

However, i will explain my backup system which is very efficient and solves basically 100% of use-cases without adding any additional complexity to the mounting system. I think a system like this is probably an even better direction to go for efficient backups:

Backup the entire TemplateVM as that is just system files and it almost never changes. There is no meaningful cache churn, etc. This is OPTIONAL because after-all this data is already all backed up in the Qubes repositories. Not restoring the system files and redownloading them is probably safer anyhow.
Utilize Borg backup in a backup VM that mounts AppVM user volume directories and backs up all user data.

This method works across all operating systems, freedesktop or not compliant systems, and creates very efficient backups.

tasket commented 6 years ago

The practice of having a backupVM that mounts appVM volumes is risky, though. And IIRC, Borg is based on scanning and comparing file data, which may be space efficient but can be heavy on CPU and IO.

This issue is aimed at improving the efficiency of dom0-based backup tools.

As for non-compliant apps (the location of .cache), nothing is perfect and the solution doesn't have to be, as long as most cache data in most situations can be left out. Of course, if users identify important exceptions they can symlink or bind the app's cache location to the standard one.

t4777sd commented 6 years ago

Borg scans files, but it can also backup from a block device. So, the .cache solution you propose would be pretty great because then borg users can gain the benefits of dom0-like security in their backups by backing up the entire block device in their BackupVM without having to mount it which avoids a whole class of security vulns.

I hope it can make it into 4.1

Btw, I looked at your script and you might want to consider borg as the backup engine. For no-cost you will gain incremental, encryption, deduplication, purging, and remote backup capabilities. To backup a block device you could use something like this:

dd if=/dev/xxx | borg create repo::disk -

DemiMarie commented 6 years ago

@t4777sd Borg’s cryptography is broken.

t4777sd commented 6 years ago

@DemiMarie I could not find anything in google about that. Can you link me to an article or something that discusses how it is broken so I can educate myself?

DemiMarie commented 6 years ago

I will give a longer answer later, but basically a malicious server can fool Borg into reusing nonces, because Borg fails to regenerate a fresh AES key every time it is run.

On Tue, Aug 21, 2018, 10:18 PM t4777sd notifications@github.com wrote:

@DemiMarie https://github.com/DemiMarie I could not find anything in google about that. Can you link me to an article or something that discusses how it is broken so I can educate myself?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4217#issuecomment-414885240, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGWB-ecSicR64E6Hwl7G3E9mn6sfHkbks5uTL91gaJpZM4V-jaK .

t4777sd commented 6 years ago

@DemiMarie, I think this was fixed by this commit: https://github.com/borgbackup/borg/pull/1374

marmarek commented 5 years ago

I can see why this may be useful, but my main concern with this feature is maintenance of increasing number of volumes (recent instance: fstab got out of sync in archlinux). It's even worse when installing a system manually in HVM (Windows, NetBSD etc). Then you need to choose what to install where etc. In some cases even identifying disks is hard.

So, if anything, I'd prefer this to be optional. Enabled manually, or after clear indication from the qube (using https://github.com/QubesOS/qubes-issues/issues/2829?) that it does know how to handle this additional device.

DemiMarie commented 5 years ago

That doesn't solve the problem. Nonces should never be stored — the risk of reuse is too high. Instead, use something like Blake2b to generate an ephemeral subkey + nonce from a master key + 256 bit random salt.

On Thu, Aug 23, 2018, 3:23 PM t4777sd notifications@github.com wrote:

@DemiMarie https://github.com/DemiMarie, I think this was fixed by this commit: borgbackup/borg#1374 https://github.com/borgbackup/borg/pull/1374

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4217#issuecomment-415540168, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGWB_qzao1oYXoXlC0nU0Rj9-Sr9WGMks5uTwEkgaJpZM4V-jaK .

ilka-schulz commented 3 years ago

I can see why this may be useful, but my main concern with this feature is maintenance of increasing number of volumes [...] So, if anything, I'd prefer this to be optional.

Additionally, even if it works fine I would consider this unexpected behavior. When I use a fancy backup tool like borg inside a running system it may skip cache directories. But when I backup an entire VM I expect it to make a complete "snapshot".

DemiMarie commented 3 years ago

I believe this should be supported, and it should be optional.

tlaurion commented 1 year ago

Random app qube opened, where persistence is needed. Machine used for safe browsing untrusted browsing. Not a main qubes.

Extract of du -chs ~/.cache/* | sort -h

9.6M    /home/user/.cache/kdenlive-thumbs.kcache
11M /home/user/.cache/icon-cache.kcache
11M /home/user/.cache/pip
11M /home/user/.cache/tracker
36M /home/user/.cache/thumbnails
416M    /home/user/.cache/chromium
1.2G    /home/user/.cache/mozilla
2.0G    total

I am not aware of any use case where backups of those cache files would be useful for anything anything? But as you can see above, are taking a lot of unneeded space and once again require users to manage that consumed space themselves. For each backup of that qube. were differential backup systems (like wyng) would not be able to deduplicate it, but would simply add and compress the differences between snapshots backuped on what is the "private" volume of that untrusted qube at the time of backup, dealing with the snapshot content to determine what changed and save it. That content is totally useless. And as later on debated, could be added through wyng (same backup timestamp if it was done today under 0.3 or 0.4 branch) and would work exactly the same way unde qubes backup today if cache volumes were taken into consideration.

Answers to past considerations:

@DemiMarie said:

I believe this should be supported, and it should be optional.

I disagree. I think it must be supported, while backing it up should be optional.

@ilka-schulz said:

Additionally, even if it works fine I would consider this unexpected behavior. When I use a fancy backup tool like borg inside a running system it may skip cache directories. But when I backup an entire VM I expect it to make a complete "snapshot".

I disagree. I think it must be seperated, while backing up that useless stuff should be optional. As of today, you do not backup volatile volumes that are part of applied on top of / (root) Template changes for obvious reasons. There are obscure cases where .cache would be needed (forensic) but outside of those use cases, those directories need to be manually removed when dealing with wyng today, otherwise accepting that a lot of space will be lost over time since we are talking of incremental backups. Even if deduplication can be applied for data that is exactly the same thing across templates and app qubes, thse cache directory will uselessly be the same.

And when doing multiple qubes backup today, the space occupied with cache is a big useless chunk of wasted space, not deduplicated across app qubes.

Reminder on how qubes rotates snapshots today (usable directly through `qvm-volume revert QubeName:VolumeName`)

Let's recheck the following before going farther (important): https://www.qubes-os.org/doc/templates/#inheritance-and-persistence QubeName: can be App Qube, TemplateVM, Standalone.... VolumeName: can be root/private today. Would be root/private/cache once this is implemented.

Qubes provides a nice tool for reverting states. Default configuration keeps 2 rollbacks possible. Qubes backup tool takes the non-alive, aka non-booted state (appqube:private, template:root+private) into its backups, and warns the user when doing backups if a qube/template is booted, warning that older saved state will be backed up (appqube:private, template:root+private, not snap volumes, read below). No problem there.

Let's take the untrusted app qube I barely use for things like this.

Those are untrusted appqube thin lvms exposed to dom0 prior of launching the qube:

[user@dom0 ~]$ ls -al /dev/mapper/qubes_dom0-vm--untrusted--private*
lrwxrwxrwx 1 root root 9 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private -> ../dm-103
lrwxrwxrwx 1 root root 8 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private--1664984990--back -> ../dm-15
lrwxrwxrwx 1 root root 8 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private--1665185704--back -> ../dm-63

We see two back states, with their "epoch" timestamps relative to the time they were created.

Let's start untrusted app qube: A new "*snap" thin lvm was created, which contains the changes that are happening under untrusted as I write those lines:

[user@dom0 ~]$ ls -al /dev/mapper/qubes_dom0-vm--untrusted--private*
lrwxrwxrwx 1 root root 9 Oct 13 11:52 /dev/mapper/qubes_dom0-vm--untrusted--private -> ../dm-103
lrwxrwxrwx 1 root root 8 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private--1664984990--back -> ../dm-15
lrwxrwxrwx 1 root root 8 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private--1665185704--back -> ../dm-63
lrwxrwxrwx 1 root root 9 Oct 13 11:52 /dev/mapper/qubes_dom0-vm--untrusted--private--snap -> ../dm-157

And now I shutdown untrusted.

Then:

user@dom0 ~]$ ls -al /dev/mapper/qubes_dom0-vm--untrusted--private*
lrwxrwxrwx 1 root root 9 Oct 13 11:55 /dev/mapper/qubes_dom0-vm--untrusted--private -> ../dm-157
lrwxrwxrwx 1 root root 8 Oct 11 14:49 /dev/mapper/qubes_dom0-vm--untrusted--private--1665185704--back -> ../dm-63
lrwxrwxrwx 1 root root 9 Oct 13 11:55 /dev/mapper/qubes_dom0-vm--untrusted--private--1665676498--back -> ../dm-103

Do we see what happened here? Recap.

Before:

Our private volume is: dm-103
Our oldest restorable state is under: dm-15
Our most recent restorable state is under: dm-63

While running:

Our private volume is: dm-103
Our oldest restorable state is under: dm-15
Our most recent restorable state is under: dm-63
Our changes are hapening under snap: dm-157

After shutdown:

Our private volume is: dm-157
Our oldest restorable state is under: dm-63
Our most recent restorable state is under: dm-103 Also:
dm-157 got deleted
timestamps of snapshots changed, not their volumes which just moved around

Reminder of how cache snapshots would behave:

Exactly the same as above.

As of today we have "back" and "snap" volumes for app qubes states (private volumes).

Those exact snapshots would exist as well for cache, and be revertible just as equally and be backed up just as easily, while private volumes would just be cleaner for everything.

Plus: experienced users could basically simply wipe such filesystems when pool space is becoming problematic. Later on even giving an option through the Qubes manager to wipe those for all qubes if we wanted.

Clarifications needed

@marmarek I'm not sure I understand the reference at https://github.com/QubesOS/qubes-issues/issues/4217#issuecomment-425450527 ?

@marmarek Agreed that Windows would probably not get distinct cache mountpoint just like it is possible to do for other TemplateVMs Qubes has direct control over today, but that could also change in the future and I'm not sure it's a proper thinking frame to limit the implementation of this needed feature, specially for https://github.com/QubesOS/qubes-issues/issues/858

tlaurion commented 1 year ago

I think it is important to distinguish having cache as volatile/having cache persistent but as a seperate filesystem layer into app qubes/have that cache made optional for backup.

@rudd-o comment at https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-451928839

Whether you go with LVM or a WAFL file system, another solution to explore would probably involve injecting into VMs a third block device (100% volatile) to mount over ~/.cache, and the backup tool explicitly excluding that volatile block device.

@marmarek : I would love to see the cache volatile (optional) on Qubes settings. But I am not sure that is what we are talking about here and the OP implementation request.

From my understanding:

Having cache layer created by default for new created app qubes
Having cache taken into consideration by default if existing for app qubes in qubes backup/qube restore tool

Later:

Having setting into qubes backup/qubes restore to deactivate cache backup/restore
Having option to wipe cache per qubes/for all all qubes
Have the cache configurable to be volatile

@tasket : comments?

tasket commented 1 year ago

For this purpose, I think Qubes' pattern of snapshot management is less important than the fact it does it at all. A '-cache' volume set can be kept synchronized to '-private' counterpart without additional planning on our part, the former just has be included in the management operations of the latter.

Pool space considerations shouldn't be affected. If a user needs cache to use less space, clearing cache files has the same effect with or without separate cache volume.

As for default behavior, that would be no '-cache' volume at all as Marek indicated. Creation and snapshotting of '-cache' would depend on VM startup code in dom0 checking a boolean VM prefs setting that can be selected by the user. The guest OS would also need to check for the presence of the cache volume, mount and link it at the same time '-private' is mounted. As a precaution, the relevant .cache paths should be purged of files if the cache pref is set. I think that's about it.

The biggest technical question may be how to select '-cache' volume size. I'm not sure there's any advantage to making it smaller than '-private'.

marmarek commented 1 year ago

The biggest technical question may be how to select '-cache' volume size. I'm not sure there's any advantage to making it smaller than '-private'.

Space management is also another issue here. With default private size of 2GB, another 2GB for .cache probably makes some sense. But if enlarge private to say 200GB, then another 200GB for cache? There are two sides for this:

if you create 200GB cache volume, the qube will be able to actually use that space; when you set private volume limit to 200GB, you probably wanted to give that qube 200GB, not 400GB
if you run out of space in the cache volume, that's yet another volume to consider when extending space; the current root/private split already cause some confusion (for example when you run out of space in something that's part of the system volume, in template-based qube); adding yet another volume would make it worse

tasket commented 1 year ago

if you run out of space in the cache volume, that's yet another volume to consider when extending space

Being an opt-in feature, maybe confusion isn't a big factor? But I would consider making the pref setting for this an integer representing megabytes, instead of a boolean. Or, the pref could be a percentage of '-private' size.

On Edit:

A think having a comment in the docs that enabling cache volume incurs extra potential for guest software to consume disk space; that would cover it.

marmarek commented 1 year ago

Being an opt-in feature, maybe confusion isn't a big factor? But I would consider making the pref setting for this an integer representing megabytes, instead of a boolean. Or, the pref could be a percentage of '-private' size.

A think having a comment in the docs that enabling cache volume incurs extra potential for guest software to consume disk space; that would cover it.

Fair enough.

QubesOS / qubes-issues