search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 926 forks source link

Moving instances between projects can require double storage capacity #10590

Open jsimpso opened 2 years ago

jsimpso commented 2 years ago

Required information

In a circumstance where the size of a VM's disk is larger than the amount of available space in the LXD storage backend, trying to move that VM between projects results in LXD trying to make a full copy of the VM and failing:

jsimpso@alcazar:~$ lxc storage list
+---------+--------+------------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |  STATE  |
+---------+--------+------------------------------------------------+-------------+---------+---------+
| default | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 6       | CREATED |
+---------+--------+------------------------------------------------+-------------+---------+---------+

jsimpso@alcazar:~$ df -h /var/snap/lxd/common/lxd/storage-pools/default
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       110G   99G  4.7G  96% /

jsimpso@alcazar:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img
-rw-r--r-- 1 root root 56G Jun 22 11:17 /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img

jsimpso@alcazar:~$ lxc move alpha alpha --project maas --target-project default
Error: Migration operation failure: Create instance from copy: Failed to run: nice -n19 dd if=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img of=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/alpha/root.img bs=16M conv=nocreat iflag=direct oflag=direct: dd: error writing '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/alpha/root.img': No space left on device
652+0 records in
651+0 records out
10921967616 bytes (11 GB, 10 GiB) copied, 78.3542 s, 139 MB/s

If other VMs are attempting to write to that storage backend when it fills up, they appear to stall, and LXD loses communication with them. Once the qemu process has been killed, LXD can re-launch the VM as expected.

Steps to reproduce

  1. Have two projects present so that we have something to move between

    jsimpso@alcazar:~$ lxc project list
+----------------+--------+----------+-----------------+----------+-------------------------+---------+
|      NAME      | IMAGES | PROFILES | STORAGE VOLUMES | NETWORKS |       DESCRIPTION       | USED BY |
+----------------+--------+----------+-----------------+----------+-------------------------+---------+
| default        | YES    | YES      | YES             | YES      | Default LXD project     | 3       |
+----------------+--------+----------+-----------------+----------+-------------------------+---------+
| maas (current) | NO     | YES      | NO              | NO       | Project managed by MAAS | 5       |
+----------------+--------+----------+-----------------+----------+-------------------------+---------+

  2. Having a storage backend with relatively little storage will make for easier testing, I tested with a 120GB disk.

  3. Initialise a few VMs, give one a bigger disk:

    lxc init ubuntu/jammy alpha --vm
ize=60GB
lxc start alpha
lxc launch ubuntu/jammy beta --vm
lxc launch ubuntu/jammy gamma --vm
lxc launch ubuntu/jammy delta --vm

  4. Grow the VM disk:

    lxc exec alpha -- dd if=/dev/zero of=anchor.file bs=1M count=50000 status=progress

    You should now have a VM whose disk is larger than the amount of available store in the backend:

    jsimpso@alcazar:~$ df -h /var/snap/lxd/common/lxd/storage-pools/default
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       110G   61G   43G  59% /

jsimpso@alcazar:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img
-rw-r--r-- 1 root root 56G Jun 22 09:00 /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img

  5. Open a connection to each running VM (lxd exec <vm> bash)

  6. On two VMs, start slowly writing to disk. Then, move the large VM between projects:

    root@beta:~# dd if=/dev/zero of=anchor.file bs=1 count=5000000000000 status=progress
root@delta:~# dd if=/dev/zero of=anchor.file bs=1 count=5000000000000 status=progress
jsimpso@alcazar:~$ lxc move alpha alpha --project maas --target-project default

During the move, the two VMs attempting to write to disk fully unresponsive, but the last one continues to respond to input.

We can then see that the qemu process is still running, but LXD has lost contact with it:

jsimpso@alcazar:~$ lxc move alpha alpha --project maas --target-project default
Error: Migration operation failure: Create instance from copy: Failed to run: nice -n19 dd if=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/maas_alpha/root.img of=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/alpha/root.img bs=16M conv=nocreat iflag=direct oflag=direct: dd: error writing '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/alpha/root.img': No space left on device
652+0 records in
651+0 records out
10921967616 bytes (11 GB, 10 GiB) copied, 78.3542 s, 139 MB/s

jsimpso@alcazar:~$ lxc ls
+-------+---------+-------------------------+------+-----------------+-----------+
| NAME  |  STATE  |          IPV4           | IPV6 |      TYPE       | SNAPSHOTS |
+-------+---------+-------------------------+------+-----------------+-----------+
| alpha | STOPPED |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| beta  | STOPPED |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| delta | STOPPED |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| gamma | RUNNING | 192.168.37.166 (enp5s0) |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+

jsimpso@alcazar:~$ lxc start beta
Error: Failed cleaning config drive mount path "/var/snap/lxd/common/lxd/devices/maas_beta/config.mount": Failed unmounting "/var/snap/lxd/common/lxd/devices/maas_beta/config.mount": Failed to unmount '/var/snap/lxd/common/lxd/devices/maas_beta/config.mount': device or resource busy
Try `lxc info --show-log beta` for more info
jsimpso@alcazar:~$ lxc info --show-log beta
Name: beta
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2022/06/22 11:11 AWST
Last Used: 2022/06/22 11:11 AWST
Error: open /var/snap/lxd/common/lxd/logs/maas_beta/qemu.log: no such file or directory

jsimpso@alcazar:~$ ps aux |  grep beta
root        9911  0.0  0.0  79904  3424 ?        Ssl  11:11   0:00 /snap/lxd/23037/bin/virtiofsd --fd=3 -o source=/var/snap/lxd/common/lxd/devices/maas_beta/config.mount
lxd         9927 19.4  7.6 1934532 614632 ?      Sl   11:11   2:27 /snap/lxd/23037/bin/qemu-system-x86_64 -S -name beta -uuid 159e794b-17e6-4658-869e-73f996996395 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/maas_beta/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/maas_beta/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/maas_beta/qemu.pid -D /var/snap/lxd/common/lxd/logs/maas_beta/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd
root        9930  0.0  0.1 1211832 13584 ?       Sl   11:11   0:00 /snap/lxd/23037/bin/virtiofsd --fd=3 -o source=/var/snap/lxd/common/lxd/devices/maas_beta/config.mount
jsimpso    10698  0.0  0.2 1384848 17496 pts/1   Sl+  11:14   0:00 lxc exec beta bash

Killing the process returns the console:

jsimpso@alcazar:~$ sudo kill 9927
jsimpso@alcazar:~$
----
root@beta:~# dd if=/dev/zero of=anchor.file bs=1 count=5000000000000 status=progress
16908336 bytes (17 MB, 16 MiB) copied, 61 s, 277 kB/sError: read vsock vm(4294967295):1873106686->vm(31):8443: connection reset by peer
jsimpso@alcazar:~$

And the VM can be started up again:

jsimpso@alcazar:~$ lxc ls
+-------+---------+-------------------------+------+-----------------+-----------+
| NAME  |  STATE  |          IPV4           | IPV6 |      TYPE       | SNAPSHOTS |
+-------+---------+-------------------------+------+-----------------+-----------+
| alpha | STOPPED |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| beta  | RUNNING |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| delta | STOPPED |                         |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
| gamma | RUNNING | 192.168.37.166 (enp5s0) |      | VIRTUAL-MACHINE | 0         |
+-------+---------+-------------------------+------+-----------------+-----------+
jsimpso@alcazar:~$
stgraber commented 2 years ago

So the fact that LXD attempts the migration is normal. We never try to guess the disk space and instead just go ahead and if we're going to hit ENOSPC, then that's going to be our cue that there isn't enough space.

The reason for that is that different storage backends store things very very differently. You can easily do cat /dev/null > out.img or the like on a VM stored on a dir storage backend, make it go to 100GiB, then move it to a project that uses a zfs storage pool and ZFS will happily store that in something like 500MiB thanks to the default inline compression.

Now, these days a project move shouldn't actually cause any data to be copied, so we'd need to check on current LXD to see what's going on there. It's also obviously problematic if QEMU just crashes or stops responding in such a situation when it itself isn't supposed to be writing anything to disk.

tomponline commented 2 years ago

Yeah seems like the target project flag is confusing the same pool move detection somehow.

tomponline commented 1 year ago

Right, so I've taken a look at whats going on here, and the behaviour is that when using instances on the dir pool the storage subsystem invokes CreateInstanceFromCopy, this then detects that the source and target are in the same pool and switches to "same-pool mode" which then invokes CreateVolumeFromCopy on the storage driver.

In the case of a storage driver that supports optimized copies (snapshots) this is very quick and doesn't take much disk space as the copy is just a snapshot of the source instance (and then in the case of ZFS, the source instance volumes are kept around but hidden as they are still referenced by the new instance's name).

In the case of the dir storage driver however, CreateVolumeFromCopy invokes genericVFSCopyVolume which uses the dd command to copy the VM's root disk device.

Basically the problem here is that there isn't a "move" concept in the storage subsystem (https://github.com/lxc/lxd/blob/632b3839b3388cc89df0ffe96d7c7c9134f00a7b/lxd/storage/pool_interface.go#L39-L145) and so the lxc move is just a lxc copy followed by a lxc delete (effectively).

We can see this where its implemented in the server side:

https://github.com/lxc/lxd/blob/632b3839b3388cc89df0ffe96d7c7c9134f00a7b/lxd/instance_post.go#L486-L553

So I'm going to put this down as a feature & maybe, as its not a bug per-se.