ZFS dataset cannot be deleted

[tomposmiko]

Required information

• Distribution: Ubuntu
• Distribution version: 20.04
• The output of "lxc info" or if that fails:

  # lxc info
  config: {}
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  MIICBDCCAYqgAwIBAgIRAOSG97yRD0xQLDionS0L80IwCgYIKoZIzj0EAwMwNDEc
  MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEBnZXR0
  bzMwHhcNMjEwNTExMTkzODM5WhcNMzEwNTA5MTkzODM5WjA0MRwwGgYDVQQKExNs
  aW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGdldHRvMzB2MBAGByqG
  SM49AgEGBSuBBAAiA2IABFc1Gpc5tMCvxv/lHhP5HYI7dgG6lya5DhTvi8JJn1BB
  NhbD55VJTZHE4k2Dru686BFGXCjaaVWLRSJO4zgD3QKivJBFLALLWdYNylWIkCt3
  xxGiBwm1EQHKakV8HXtTNaNgMF4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG
  CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0RBCIwIIIGZ2V0dG8zhwR/AAAB
  hxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDsnULiMXuvpPWo
  WCeeqQZmlxvweCkH3DSGnUnXS2WjQhn7D2rM45LI7RBVUJ9DD88CME+IO7XE174K
  7Tuv4JmuUWNrwm5bGbt7lG6SI8DCVr8Q9FPw99L374gabnz/NRpY9g==
  -----END CERTIFICATE-----
  certificate_fingerprint: 0e8693910d55d5b8e6da639696dbb32e8efe5d28dda2dc7fea1193d271839046
  driver: lxc | qemu
  driver_version: 4.0.11 | 6.1.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  seccomp_listener_continue: "true"
  shiftfs: "false"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 5.4.0-80-generic
  lxc_features:
  cgroup2: "true"
  core_scheduling: "true"
  devpts_fd: "true"
  idmapped_mounts_v2: "true"
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  network_veth_router: "true"
  pidfd: "true"
  seccomp_allow_deny_syntax: "true"
  seccomp_notify: "true"
  seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: getto3
  server_pid: 219915
  server_version: "4.19"
  storage: zfs
  storage_version: 2.1.0-0york2~20.04
  storage_supported_drivers:
  - name: btrfs
  version: 5.4.1
  remote: false
  - name: cephfs
  version: 15.2.14
  remote: true
  - name: dir
  version: "1"
  remote: false
  - name: lvm
  version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0
  remote: false
  - name: zfs
  version: 2.1.0-0york2~20.04
  remote: false
  - name: ceph
  version: 15.2.14
  remote: true

Issue description

Container cannot be deleted. I don't know if it matters, first I tried to do it via the -f enabled.

Error: Error deleting storage volume: Failed to run: zfs destroy -r tank/lxd/containers/vcc: cannot destroy 'tank/lxd/containers/vcc': dataset is busy

Steps to reproduce

I do not know any.

[stgraber]

grep containers/vcc /proc/*/mountinfo

That should tell us why it thinks it's busy

[tomposmiko]

root@getto3:~# grep containers/vcc /proc/*/mountinfo
/proc/216040/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
/proc/3663992/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
root@getto3:~# ps ax|grep 16040|grep -v grep
 216040 ?        Sl     0:47 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root@getto3:~# ps ax|grep 3663992|grep -v grep
3663992 ?        Ss     0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers koko

Should lxcfs exit, when container is stopped?

As for the 3663992 I don't understand, it's a totally different container. I probably don't understand the logic here well...

[stgraber]

Hmm, yeah, looks like they cloned the mount entry which is preventing the deletion.

You can likely sort it out with: nsenter -t 216040 umount /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc

[tomposmiko]

Still not good:

root@getto3:~# grep vcc /proc/*/mountinfo
/proc/216040/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
/proc/3663992/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
root@getto3:~# nsenter -t 216040 umount /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc
umount: /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc: not mounted.
root@getto3:~# grep vcc /proc/*/mountinfo
/proc/216040/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
/proc/3663992/mountinfo:2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl

[stgraber]

Can you show cat /proc/216040/mountinfo?

[tomposmiko]

Sure thing:

root@getto3:~# cat /proc/216040/mountinfo
2951 3078 0:27 / /var/lib/snapd/hostfs rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
2957 2951 0:25 / /var/lib/snapd/hostfs/run rw,nosuid,nodev,noexec,relatime master:5 - tmpfs tmpfs rw,size=810968k,mode=755
2958 2957 0:30 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
2959 2957 0:25 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=810968k,mode=755
2986 2951 0:50 / /var/lib/snapd/hostfs/tank rw,noatime master:105 - zfs tank rw,xattr,noacl
2994 2951 0:55 / /var/lib/snapd/hostfs/var/snap/lxd/common/ns rw,relatime master:710 - tmpfs tmpfs rw,size=1024k,mode=700
2995 2951 7:7 / /var/lib/snapd/hostfs/snap/core20/1081 ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3015 538 7:7 / / ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3016 3015 0:6 / /dev rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=4010756k,nr_inodes=1002689,mode=755
3017 3016 0:24 / /dev/pts rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
3018 3016 0:29 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw
3019 3016 0:21 / /dev/mqueue rw,nosuid,nodev,noexec,relatime master:27 - mqueue mqueue rw
3020 3016 0:48 / /dev/hugepages rw,relatime master:28 - hugetlbfs hugetlbfs rw,pagesize=2M
3021 3015 0:27 /etc /etc rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3022 3015 0:27 /home /home rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3023 3015 0:27 /root /root rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3024 3015 0:5 / /proc rw,nosuid,nodev,noexec,relatime master:14 - proc proc rw
3025 3024 0:47 / /proc/sys/fs/binfmt_misc rw,relatime master:26 - autofs systemd-1 rw,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18667
3026 3025 0:69 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime master:301 - binfmt_misc binfmt_misc rw
3027 3015 0:23 / /sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
3028 3027 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
3029 3027 0:31 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:9 - tmpfs tmpfs ro,mode=755
3030 3029 0:32 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime master:10 - cgroup2 cgroup2 rw
3031 3029 0:33 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd
3032 3029 0:36 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,cpu,cpuacct
3033 3029 0:37 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,devices
3034 3029 0:38 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,net_cls,net_prio
3035 3029 0:39 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,freezer
3036 3029 0:40 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,cpuset,clone_children
3037 3029 0:41 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,memory
3038 3029 0:42 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,pids
3039 3029 0:43 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,blkio
3040 3029 0:44 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,hugetlb
3041 3029 0:45 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,perf_event
3042 3029 0:46 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:25 - cgroup cgroup rw,rdma
3043 3027 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
3044 3027 0:35 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime master:13 - bpf none rw,mode=700
3045 3027 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime master:29 - debugfs debugfs rw
3046 3027 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime master:30 - tracefs tracefs rw
3047 3027 0:22 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime master:66 - configfs configfs rw
3048 3027 0:49 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime master:68 - fusectl fusectl rw
3049 3015 0:27 /tmp /tmp rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3050 3015 0:27 /var/snap /var/snap rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3051 3050 0:55 / /var/snap/lxd/common/ns rw,relatime master:710 - tmpfs tmpfs rw,size=1024k,mode=700
3052 3015 0:27 /var/lib/snapd /var/lib/snapd rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3053 3015 0:27 /var/tmp /var/tmp rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3054 3015 0:25 / /run rw,nosuid,nodev,noexec,relatime master:5 - tmpfs tmpfs rw,size=810968k,mode=755
3055 3054 0:30 / /run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
3056 3054 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=810968k,mode=755
3058 3015 0:27 /usr/lib/modules /usr/lib/modules rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3059 3015 0:27 /usr/lib/firmware /usr/lib/firmware rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3060 3015 0:27 /usr/src /usr/src rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3061 3015 0:27 /var/log /var/log rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3062 3015 0:27 /media /media rw,relatime shared:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3063 3054 0:25 /netns /run/netns rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=810968k,mode=755
3064 3015 0:27 /mnt /mnt rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3065 3021 7:7 /etc/nsswitch.conf /etc/nsswitch.conf ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3066 3021 7:7 /etc/apparmor /etc/apparmor ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3067 3021 7:7 /etc/apparmor.d /etc/apparmor.d ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3068 3015 7:2 /usr/lib/snapd /usr/lib/snapd ro,nodev,relatime - squashfs /dev/loop2 ro
3069 3015 0:27 /snap /snap rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3077 3069 7:7 / /snap/core20/1081 ro,nodev,relatime master:699 - squashfs /dev/loop7 ro
3078 3052 0:27 /var/lib/snapd/hostfs /var/lib/snapd/hostfs rw,relatime - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
2952 3049 0:27 /tmp/snap.lxd/tmp /tmp rw,relatime - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
2953 3017 0:121 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
2954 3016 0:121 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
97 3050 0:27 /var/snap/lxd/common/lxd/storage-pools /var/snap/lxd/common/lxd/storage-pools rw,relatime shared:123 master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
166 3050 0:27 /var/snap/lxd/common/lxd/devices /var/snap/lxd/common/lxd/devices rw,relatime shared:124 master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
3098 3050 0:54 / /var/snap/lxd/common/shmounts rw,relatime shared:311 - tmpfs tmpfs rw,size=1024k,mode=711
3099 3098 0:61 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:298 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
3141 3098 0:62 / /var/snap/lxd/common/shmounts/instances rw,relatime shared:299 - tmpfs tmpfs rw,size=100k,mode=711
407 3098 0:87 / /var/snap/lxd/common/shmounts/storage-pools/default/containers/ns-ch14 rw,noatime shared:288 - zfs tank/lxd/containers/ns-ch14 rw,xattr,posixacl
2961 97 0:64 / /var/snap/lxd/common/lxd/storage-pools/default/containers/gitanes rw,noatime shared:179 - zfs tank/lxd/containers/gitanes rw,xattr,posixacl
226 97 0:76 / /var/snap/lxd/common/lxd/storage-pools/default/containers/koko rw,noatime shared:238 - zfs tank/lxd/containers/koko rw,xattr,posixacl
232 97 0:93 / /var/snap/lxd/common/lxd/storage-pools/default/containers/proxy rw,noatime shared:295 - zfs tank/lxd/containers/proxy rw,xattr,posixacl
277 97 0:107 / /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/pts rw,noatime shared:296 - zfs tank/lxd/virtual-machines/pts rw,xattr,posixacl
357 97 0:109 / /var/snap/lxd/common/lxd/storage-pools/default/containers/wifi rw,noatime shared:385 - zfs tank/lxd/containers/wifi rw,xattr,posixacl
476 3098 0:120 / /var/snap/lxd/common/shmounts rw,relatime shared:300 - tmpfs tmpfs rw,size=1024k,mode=711
647 3015 7:5 /wrappers/kmod /usr/bin/kmod ro,nodev,relatime - squashfs /dev/loop5 ro
752 3015 0:27 /usr/share/ca-certificates /usr/share/ca-certificates rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
786 2951 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=4010756k,nr_inodes=1002689,mode=755
1102 2951 0:5 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:14 - proc proc rw
1502 2951 0:23 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
2962 476 0:127 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:428 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2985 3050 0:128 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
2915 97 0:126 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc rw,noatime shared:114 - zfs tank/lxd/containers/vcc rw,xattr,posixacl
92 3015 0:27 /boot /boot rw,relatime master:1 - btrfs /dev/mapper/vg0-root rw,ssd,discard,space_cache,subvolid=5,subvol=/
93 3054 0:122 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
161 3021 0:123 / /etc rw,relatime - tmpfs tmpfs rw,mode=755
162 3015 0:124 / /usr/share/misc rw,relatime - tmpfs tmpfs rw,mode=755
404 2951 7:10 / /var/lib/snapd/hostfs/snap/core20/1169 ro,nodev,relatime master:108 - squashfs /dev/loop10 ro
221 3069 7:10 / /snap/core20/1169 ro,nodev,relatime master:108 - squashfs /dev/loop10 ro
3905 2951 7:12 / /var/lib/snapd/hostfs/snap/snapd/13270 ro,nodev,relatime master:975 - squashfs /dev/loop12 ro
3879 3069 7:12 / /snap/snapd/13270 ro,nodev,relatime master:975 - squashfs /dev/loop12 ro
2670 476 0:140 / /var/snap/lxd/common/shmounts rw,relatime shared:235 - tmpfs tmpfs rw,size=1024k,mode=711
2955 2670 0:147 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:243 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
3004 2670 0:148 / /var/snap/lxd/common/shmounts/instances rw,relatime shared:688 - tmpfs tmpfs rw,size=100k,mode=711
3849 2951 7:13 / /var/lib/snapd/hostfs/snap/snapd/13640 ro,nodev,relatime master:921 - squashfs /dev/loop13 ro
3837 3069 7:13 / /snap/snapd/13640 ro,nodev,relatime master:921 - squashfs /dev/loop13 ro
1456 2951 7:9 / /var/lib/snapd/hostfs/snap/lxd/21750 ro,nodev,relatime master:128 - squashfs /dev/loop9 ro
471 3069 7:9 / /snap/lxd/21750 ro,nodev,relatime master:128 - squashfs /dev/loop9 ro
3809 2951 7:6 / /var/lib/snapd/hostfs/snap/lxd/21780 ro,nodev,relatime master:897 - squashfs /dev/loop6 ro
3797 3069 7:6 / /snap/lxd/21780 ro,nodev,relatime master:897 - squashfs /dev/loop6 ro
2648 3054 0:129 / /run/user/1000 rw,nosuid,nodev,relatime master:178 - tmpfs tmpfs rw,size=810968k,mode=700,uid=1000,gid=1000
2647 2957 0:129 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:178 - tmpfs tmpfs rw,size=810968k,mode=700,uid=1000,gid=1000

[stgraber]

Doh, it was just a typo earlier, the following should work:

nsenter -t 216040 -m umount /var/snap/lxd/common/lxd/storage-pools/default/containers/vcc

[tomposmiko]

Yeah, that did the trick, thank you. So is this a kind of bug?

[stgraber]

Yeah, it's a very mild version of the same problem you reported in https://github.com/lxc/lxd-pkg-snap/issues/61

[stgraber]

(Closing as we have the other bug for tracking)

[tomposmiko]

@stgraber Are you sure about this? This time I was able to start the container after stopping it with no problem.

[stgraber]

That's because a few weeks ago we made a change to LXD which allows it to bypass the mount check in ZFS. The underlying problem is still the same but was effectively made less visible.

[tomposmiko]

Alright, thanks, that makes it a bit more usable.