search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

Can't remove container after update from 3.17 -> 3.18 #6483

Closed forest-official closed 4 years ago

forest-official commented 4 years ago

Box

root@beta-0002:/var/snap/lxd/common/shmounts# uname -a
Linux beta-0002 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@beta-0002:/var/snap/lxd/common/shmounts# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic
root@beta-0002:/var/snap/lxd/common/shmounts# snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.42.1  8039   stable    canonical✓  core
lxd   3.18       12317  3.18      canonical✓  -

Problem

When i try to delete a container, i get the following error.

Error: Failed to destroy ZFS dataset: Failed to run: zfs destroy -r forest/lxd/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e: umount: /var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e: mountpoint not found
cannot unmount '/var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e': umount failed

It seems lxd expects mounts to reside in /var/snap/lxd/common/shmounts/storage-pools/zfs/containers, while containers seem to be mounted in /var/snap/lxd/common/lxd/storage-pools/zfs/containers

I have tried to create the directory & mount the zfs volume there:

root@beta-0002:/var/snap/lxd/common/shmounts# mkdir -p storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e
root@beta-0002:/var/snap/lxd/common/shmounts# zfs set mountpoint=/var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e forest/lxd/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e
root@beta-0002:/var/snap/lxd/common/shmounts# zfs mount forest/lxd/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e

Still, i'm getting the same error:

root@beta-0002:/var/snap/lxd/common/shmounts# lxc delete tree-cb70ae54-509d-44d1-9994-10d02abba37e
Error: Failed to destroy ZFS dataset: Failed to run: zfs destroy -r forest/lxd/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e: umount: /var/snap/lxd/common/shmounts/storage-poo
ls/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e: mountpoint not found
cannot unmount '/var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e': umount failed

While i'm able to unmount it myself:

root@beta-0002:/var/snap/lxd/common/shmounts# l /var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e/
backup.yaml  metadata.yaml  rootfs/  templates/
root@beta-0002:/var/snap/lxd/common/shmounts# umount !$
umount /var/snap/lxd/common/shmounts/storage-pools/zfs/containers/tree-cb70ae54-509d-44d1-9994-10d02abba37e/

Is there some way i can work around this?

stgraber commented 4 years ago

What's that system's uptime?

Logic around mounts has been changed in the snap packaging around October 20th to avoid such issues, so if the system hasn't been rebooted since then, I would strongly recommend that you do, this should take care of this issue.

forest-official commented 4 years ago

The system has been running for quite some time, i'll reboot the cluster at a more convenient time and report back.

stgraber commented 4 years ago

Ok, tentatively closing assuming that the fixes we've done around shmounts/shared in snap has fixed this issue. Let us know if that's still happening after reboot.