search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

Require size.state to be greated than memory for stateful stop/snapshot #9723

Closed vosdev closed 2 years ago

vosdev commented 2 years ago

Required information

Issue description

I evacuated a node for a reboot and restored it after it came back online.

~
root @ node1 # lxc cluster restore node3
Are you sure you want to restore cluster member "node3"? (yes/no) [default=no]: yes
Error: Failed to start instance "k8s-dev": write /var/snap/lxd/common/lxd/virtual-machines/k8s-dev/config/lxd-agent: copy_file_range: no space left on device

the VM k8s-dev lives on ceph. 

~
root @ node1 # rbd info LXD/virtual-machine_k8s-dev
rbd image 'virtual-machine_k8s-dev':
        size 95 MiB in 24 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 1e5d2e267b5d41
        block_name_prefix: rbd_data.1e5d2e267b5d41
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Tue Aug  3 15:15:33 2021
        access_timestamp: Tue Aug  3 15:15:33 2021
        modify_timestamp: Tue Aug  3 15:15:33 2021
        parent: LXD/zombie_image_b1f0967a5c36cf51627a1e99f516c4612fc7ec5595e26cfef43d3e6aca06f35f_ext4@readonly
        overlap: 95 MiB

~
root @ node1 # rbd info LXD/virtual-machine_k8s-dev.block
rbd image 'virtual-machine_k8s-dev.block':
        size 19 GiB in 4769 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 1e5d284f63913a
        block_name_prefix: rbd_data.1e5d284f63913a
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Tue Aug  3 15:15:33 2021
        access_timestamp: Tue Aug  3 15:15:33 2021
        modify_timestamp: Tue Aug  3 15:15:33 2021
        parent: LXD/zombie_image_b1f0967a5c36cf51627a1e99f516c4612fc7ec5595e26cfef43d3e6aca06f35f_ext4.block@readonly
        overlap: 19 GiB

As you can see, the config image on ceph is 100% full because of the ./state file

~
root @ node1 # rbd -p LXD map virtual-machine_k8s-dev
/dev/rbd4

~
root @ node1 # mount /dev/rbd4 /mnt

~
root @ node1 # df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd4        89M   87M     0 100% /mnt

~
root @ node1 # du -sh /mnt/*
4.0K    /mnt/agent-client.crt
4.0K    /mnt/agent-client.key
4.0K    /mnt/agent.crt
4.0K    /mnt/agent.key
4.0K    /mnt/backup.yaml
10M     /mnt/config
4.0K    /mnt/config.mount
16K     /mnt/lost+found
4.0K    /mnt/metadata.yaml
128K    /mnt/qemu.nvram
77M     /mnt/state
28K     /mnt/templates

Additional information:

this VM, k8s-dev, is the first VM on this node. 3 other instances were containers and started without issue.

Another VM that lives on this host+ceph that has not started yet only uses 12% of the config image on ceph and has no ./state file

The difference between the two VMs is that k8s-dev has migration.stateful: "true" in it's config and the other VMs/containers do not.

I used this VM as a test for the new stateful migration feature, but never got it to work. The command would just wait indefinitely. I forgot about it until now. The state file is from November 17th, a little after VM live migration came available and I started testing it.

-rw-r--r-- 1 root root 79880192 Nov 17 09:47 /mnt/state

Is LXD writing the VMs memory to the ceph config image to transfer it to another host? If so, then a 100mb quota isn't going to be enough?

Information to attach

Name: k8s-dev
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Location: node3
Created: 2021/08/03 15:15 CEST
Last Used: 2021/11/24 08:42 CET
Error: open /var/snap/lxd/common/lxd/logs/k8s-dev/qemu.log: no such file or directory
architecture: x86_64
config:
  boot.autostart.delay: "3"
  boot.host_shutdown_timeout: "90"
  cluster.evacuate: stop
  image.architecture: amd64
  image.description: Ubuntu focal amd64 (20210802_07:42)
  image.os: Ubuntu
  image.release: focal
  image.serial: "20210802_07:42"
  image.type: disk-kvm.img
  image.variant: cloud
  limits.cpu: "6"
  limits.memory: 4GB
  limits.memory.enforce: soft
  migration.stateful: "true"
  user.user-data: |
    #cloud-config
    packages:
      - vim
      - htop
      - facter
      - curl
      - ssh
    users:
      - name: ansible
        groups: sudo
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NYQMZN Ansible Automation 08-2019
        sudo: ALL=(ALL) NOPASSWD:ALL
  volatile.base_image: b1f0967a5c36cf51627a1e99f516c4612fc7ec5595e26cfef43d3e6aca06f35f
  volatile.eth0.hwaddr: 00:16:3e:01:21:d7
  volatile.last_state.power: RUNNING
  volatile.uuid: c312ef06-77e9-4ea2-aced-bb26bfb4afa2
  volatile.vsock_id: "156"
devices:
  eth0:
    nictype: bridged
    parent: br121
    type: nic
  root:
    path: /
    pool: ceph
    type: disk
ephemeral: false
profiles:
- default
- limits.medium
- cloud-init
stateful: false
description: ""

The last entry of /var/snap/lxd/common/lxd/logs/lxd.log is that it succesfully started the previous container from the lxc cluster recover action, so not relevant.

vosdev commented 2 years ago

For now, can I safely remove the state file to get my VMs/containers to restore using the lxc cluster restore command?

stgraber commented 2 years ago

Right, so your stateful migration never failed, instead it was stuck on I/O due to lack of disk space. Then on restart, the instance config disk is so full that LXD can't actually start the instance back up.

Your best bet is to do lxc config device set INSTANCE root size.state 8GiB or something along those lines which will allow for enough space for stateful stop/snapshots/migration and also fix your current startup problem.

vosdev commented 2 years ago

That is a critical oversight on my part :-). It even says so on your newspost! https://discuss.linuxcontainers.org/t/lxd-4-12-has-been-released/10424

Can I request we get an error + automatic clean-up for when this happens instead of a stuck process + a no space left on device at next restart of the VM? Or an initial "If memory > diskspace" then do not even attempt to transfer state

I just changed the size.state device and the VM works now

~
root @ node1 # lxc config device override k8s-dev root
Device root overridden for k8s-dev

~
root @ node1 # lxc config device set k8s-dev root size.state 8GiB

Cheers :-). Now I can also start properly testing/using the live migration feature!

edit: The information was only mentioned in the release notes of 4.12, not 4.20. I have also been unable to find it on the docs

stgraber commented 2 years ago

Yeah, I think it'd be reasonable for us to refuse performing stateful stop/snapshots/migration unless size.state is >= size + limits.memory.

tomponline commented 2 years ago

Yes that would make it more user friendly to discover that setting.

stgraber commented 2 years ago

I think we should do that on startup instead of during config validation as this needs to check:

Mixing that in with profiles and the like could cause a lot of config update failures, so probably best to just fail startup by validating this in Start() of driver_qemu.go