Users frequently run into dom0 storage pool capacity problems, especially when installing templates

[jpds]

Qubes OS version 4.1

Affected component(s) or functionality Gentoo template

Brief summary I had qubes-template-gentoo installed from the -testing repo from months ago. I decided to uninstall it, and see if there was a newer version available from qubes-templates-community, when installing this happened:

$ sudo qubes-dom0-update --enablerepo=qubes-templates-community qubes-template-gentoo
Using sys-vpn as UpdateVM to download updates for Dom0; this may take some time...
Last metadata expiration check: 0:49:09 ago on Mon Dec 21 13:48:29 2020.
Dependencies resolved.
================================================================================
 Package              Arch   Version            Repository                 Size
================================================================================
Installing:
 qubes-template-gentoo
                      noarch 4.0.6-202009261145 qubes-templates-community 1.2 G

Transaction Summary
================================================================================
Install  1 Package

Total size: 1.2 G
Installed size: 7.5 G
DNF will only download packages for the transaction.
Downloading Packages:
[SKIPPED] qubes-template-gentoo-4.0.6-202009261145.noarch.rpm: Already downloaded
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Qubes OS Repository for Dom0                                                                       2.9 MB/s | 3.0 kB     00:00    
Qubes OS Repository for Dom0                                                                       585 kB/s | 1.1 kB     00:00    
Dependencies resolved.
===================================================================================================================================
 Package                             Architecture         Version                            Repository                       Size
===================================================================================================================================
Installing:
 qubes-template-gentoo               noarch               4.0.6-202009261145                 qubes-dom0-cached               1.2 G

Transaction Summary
===================================================================================================================================
Install  1 Package

Total size: 1.2 G
Installed size: 7.5 G
Is this ok [y/N]: Y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Error: Transaction test error:
  installing package qubes-template-gentoo-4.0.6-202009261145.noarch needs 336MB on the / filesystem

Error Summary
-------------
Disk Requirements:
   At least 336MB more space needed on the / filesystem.

That's a reasonable disk requirement, but all my partitions have more space than that:

$ df -h 
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     1.9G     0  1.9G   0% /dev
tmpfs                        2.0G     0  2.0G   0% /dev/shm
tmpfs                        2.0G  6.5M  2.0G   1% /run
/dev/mapper/qubes_dom0-root   20G   12G  7.6G  60% /
tmpfs                        2.0G  8.0K  2.0G   1% /tmp
xenstore                     2.0G  1.2M  2.0G   1% /var/lib/xenstored
/dev/nvme0n1p2               976M  280M  630M  31% /boot
/dev/nvme0n1p1               599M  6.5M  593M   2% /boot/efi
tmpfs                        392M   32K  392M   1% /run/user/1000

I have also purged the dnf cache from my dom0/UpdateVM as well.

How Reproducible Always.

Relevant documentation you've consulted

• https://www.qubes-os.org/news/2020/10/05/new-gentoo-templates-and-maintenance-infrastructure/
• https://github.com/Qubes-Community/Contents/blob/master/docs/os/gentoo.md

[andrewdavidwong]

You might just be out of space in your UpdateVM (sys-firewall by default).

[andrewdavidwong]

Installed size: 7.5 G

/dev/mapper/qubes_dom0-root 20G 12G 7.6G 60% /

So you have 0.1 G to spare (if these are even referring to the same thing)?

[fepitre]

The issue is that because the RPM is extracted in the dom0 thin pool which is only 20GB now (currently) in upcoming R4.1. It's only after extraction that lvm volumes are created in vm-pool. I would to increase the dom0 pool to few GB.

[andrewdavidwong]

The issue is that because the RPM is extracted in the dom0 thin pool which is only 20GB now (currently) in upcoming R4.1. It's only after extraction that lvm volumes are created in vm-pool. I would to increase the dom0 pool to few GB.

It sounds like the fundamental problem is that, from the user's perspective, it appears that there is enough space available, but in actuality, there is not. Is that correct?

[fepitre]

@andrewdavidwong Yes. @marmarek what should we do exactly because this is happening more and more (even for me as you know)? Should we increase dom0 to 30GB? I even thought to a separate LVM partition which could be in vm-pool and for which we mount it for updates/installs? It has the advantage to be flexible but add a possible layer of complexity.

[tlaurion]

@fepitre #1957 but a disk cache for updates? +1!

[eaon]

Heads up for folks running into this, as @rocodes pointed out in our issue: while there's some "redirects" for template installs when it comes to qubes-dom0-update, manually installing a template via dnf may still run into this. AFAICT, manually installing the same file with qvm-template will work fine though.