ubojan commented 4 years ago

Required information

Distribution: Ubuntu
Distribution version: Ubuntu 18.04.3 LTS
The output of "lxc info" or if that fails: Error: Get http://unix.socket/1.0: EOF
- Kernel version: 5.0.0-1028-gcp
- LXC version: 3.21
- LXD version: 3.21
- Storage backend in use: ZFS

Issue description

lxd version 3.18 was working fine. I've refreshed lxd snap and task failed to complete. I've tried to revert back to 3.18, that failed as well:

- Start snap "lxd" (12631) services ([start snap.lxd.activate.service] failed with exit status 1: Job for snap.lxd.activate.service failed because the control process exited with error code.
See "systemctl status snap.lxd.activate.service" and "journalctl -xe" for details.
)

(^ no useful messages which point to cause when running those two suggested commands)

Steps to reproduce

1.snap refresh

task fails to complete, lxd service keeps restarting

Information to attach

[X] Any relevant kernel output (dmesg) no relevant messages
[X] Container log (lxc info NAME --show-log) N/A
[X] Container configuration (lxc config show NAME --expanded) service is down

[X] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)

t=2020-02-19T16:07:06+0000 lvl=info msg="LXD 3.21 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2020-02-19T16:07:06+0000 lvl=info msg="Kernel uid/gid map:"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - u 0 0 4294967295"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - g 0 0 4294967295"
t=2020-02-19T16:07:06+0000 lvl=info msg="Configured LXD uid/gid map:"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - u 0 1000000 1000000000"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - g 0 1000000 1000000000"
t=2020-02-19T16:07:06+0000 lvl=info msg="Kernel features:"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - uevent injection: yes"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - seccomp listener: yes"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - unprivileged file capabilities: yes"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - cgroup layout: hybrid"
t=2020-02-19T16:07:06+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
t=2020-02-19T16:07:06+0000 lvl=info msg=" - shiftfs support: disabled"
t=2020-02-19T16:07:06+0000 lvl=info msg="Initializing local database"
t=2020-02-19T16:07:07+0000 lvl=info msg="Starting /dev/lxd handler:"
t=2020-02-19T16:07:07+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2020-02-19T16:07:07+0000 lvl=info msg="REST API daemon:"
t=2020-02-19T16:07:07+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2020-02-19T16:07:07+0000 lvl=info msg=" - binding TCP socket" socket=10.240.2.50:8443
t=2020-02-19T16:07:07+0000 lvl=info msg="Initializing global database"
t=2020-02-19T16:07:07+0000 lvl=info msg="Initializing storage pools"
t=2020-02-19T16:07:07+0000 lvl=info msg="Applying patch: storage_zfs_mount"
t=2020-02-19T16:07:08+0000 lvl=eror msg="Failed to start the daemon: Failed to run: zfs set mountpoint=/var/snap/lxd/common/lxd/storage-pools/default/containers/xxx-xxxxxxx canmount=noauto lxd/containers/xxx-xxxxxxx: umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/xxx-xxxxxxx: mountpoint not found\ncannot unmount '/var/snap/lxd/common/shmounts/storage-pools/default/containers/xxx-xxxxxxx': umount failed"
t=2020-02-19T16:07:08+0000 lvl=info msg="Starting shutdown sequence"
t=2020-02-19T16:07:08+0000 lvl=info msg="Stopping REST API handler:"
t=2020-02-19T16:07:08+0000 lvl=info msg=" - closing socket" socket=10.240.2.50:8443
t=2020-02-19T16:07:08+0000 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/unix.socket
t=2020-02-19T16:07:08+0000 lvl=info msg="Stopping /dev/lxd handler:"
t=2020-02-19T16:07:08+0000 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2020-02-19T16:07:09+0000 lvl=info msg="Closing the database"

[X] Output of the client with --debug N/A
[X] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)
```
# lxc monitor
Error: Get http://unix.socket/1.0: EOF
```

stgraber commented 4 years ago

Looks like you have some zfs mount issues going on here. If it's an option, I'd recommend rebooting the system, this will effectively cleanup your mount namespace and unstick zfs.

If that's not an option, we can try to sort out what's blocking it in place assuming there's no kernel bug in the mix.

ubojan commented 4 years ago

Thanks for your help. I've rebooted the server and that fixed the problem; lxd got updated to 3.21 and is working fine.

ubojan commented 4 years ago

@stgraber : this specific issue is happening on way too many servers where I refresh lxd snap. I would like to avoid reboots if that's possible and not too much work.

You've mentioned alternative option - to check what's blocking it.

Do you have some kind of check in mind which I could run on servers before lxd snap refresh, that could indicate this specific issue can happen in case of snap refresh?
In case if issue happens after lxd snap refresh, what could be done in a way to try and fix it without server reboot?

stgraber commented 4 years ago

nsenter --mount=/run/snapd/ns/lxd.mnt -- cat /proc/self/mountinfo

ubojan commented 4 years ago

Thanks for your response. I've ran the command on server before I've executed lxd snap refresh (which failed, so I had to reboot it). Also, I hope you don't mind that I've redacted container names.

root@lxd-server:~# nsenter --mount=/run/snapd/ns/lxd.mnt -- cat /proc/self/mountinfo
268 376 8:1 / /var/lib/snapd/hostfs rw,relatime master:1 - ext4 /dev/sda1 rw
274 268 0:23 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=12376608k,mode=755
275 274 0:25 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
276 274 0:23 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=12376608k,mode=755
314 268 7:0 / /var/lib/snapd/hostfs/snap/lxd/12211 ro,nodev,relatime master:30 - squashfs /dev/loop0 ro
315 268 7:1 / /var/lib/snapd/hostfs/snap/lxd/11824 ro,nodev,relatime master:31 - squashfs /dev/loop1 ro
316 268 7:2 / /var/lib/snapd/hostfs/snap/core/7396 ro,nodev,relatime master:32 - squashfs /dev/loop2 ro
317 268 7:3 / /var/lib/snapd/hostfs/snap/core/7917 ro,nodev,relatime master:33 - squashfs /dev/loop3 ro
318 268 0:47 / /var/lib/snapd/hostfs/var/lib/lxcfs rw,nosuid,nodev,relatime master:131 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
326 267 7:3 / / ro,nodev,relatime master:33 - squashfs /dev/loop3 ro
327 326 0:6 / /dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=61871588k,nr_inodes=15467897,mode=755
328 327 0:22 / /dev/pts rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
329 327 0:24 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw
330 327 0:41 / /dev/hugepages rw,relatime master:25 - hugetlbfs hugetlbfs rw,pagesize=2M
331 327 0:19 / /dev/mqueue rw,relatime master:27 - mqueue mqueue rw
333 326 8:1 /home /home rw,relatime master:1 - ext4 /dev/sda1 rw
334 326 8:1 /root /root rw,relatime master:1 - ext4 /dev/sda1 rw
335 326 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
336 326 0:21 / /sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
337 336 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
338 336 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:9 - tmpfs tmpfs ro,mode=755
339 338 0:27 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime master:10 - cgroup2 cgroup rw
340 338 0:28 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd
341 338 0:30 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,rdma
342 338 0:31 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,net_cls,net_prio
343 338 0:32 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,cpuset,clone_children
344 338 0:33 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,blkio
345 338 0:34 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,devices
346 338 0:35 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,cpu,cpuacct
347 338 0:36 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,perf_event
348 338 0:37 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,pids
349 338 0:38 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,memory
350 338 0:39 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,freezer
351 338 0:40 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,hugetlb
352 336 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
353 336 0:8 / /sys/kernel/debug rw,relatime master:26 - debugfs debugfs rw
354 336 0:20 / /sys/kernel/config rw,relatime master:28 - configfs configfs rw
355 336 0:42 / /sys/fs/fuse/connections rw,relatime master:29 - fusectl fusectl rw
356 326 8:1 /tmp /tmp rw,relatime master:1 - ext4 /dev/sda1 rw
357 326 8:1 /var/snap /var/snap rw,relatime master:1 - ext4 /dev/sda1 rw
358 326 8:1 /var/lib/snapd /var/lib/snapd rw,relatime master:1 - ext4 /dev/sda1 rw
359 326 8:1 /var/tmp /var/tmp rw,relatime master:1 - ext4 /dev/sda1 rw
363 326 8:1 /lib/modules /lib/modules rw,relatime master:1 - ext4 /dev/sda1 rw
364 326 8:1 /usr/src /usr/src rw,relatime master:1 - ext4 /dev/sda1 rw
365 326 8:1 /var/log /var/log rw,relatime master:1 - ext4 /dev/sda1 rw
366 326 8:1 /media /media rw,relatime shared:1 - ext4 /dev/sda1 rw
368 326 8:1 /mnt /mnt rw,relatime master:1 - ext4 /dev/sda1 rw
371 326 8:1 /snap /snap rw,relatime master:1 - ext4 /dev/sda1 rw
372 371 7:0 / /snap/lxd/12211 ro,nodev,relatime master:30 - squashfs /dev/loop0 ro
373 371 7:1 / /snap/lxd/11824 ro,nodev,relatime master:31 - squashfs /dev/loop1 ro
374 371 7:2 / /snap/core/7396 ro,nodev,relatime master:32 - squashfs /dev/loop2 ro
375 371 7:3 / /snap/core/7917 ro,nodev,relatime master:33 - squashfs /dev/loop3 ro
376 358 8:1 /var/lib/snapd/hostfs /var/lib/snapd/hostfs rw,relatime - ext4 /dev/sda1 rw
269 356 8:1 /tmp/snap.lxd/tmp /tmp rw,relatime - ext4 /dev/sda1 rw
270 328 0:48 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
271 327 0:48 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
273 357 8:1 /var/snap/lxd/common/lxd/storage-pools /var/snap/lxd/common/lxd/storage-pools rw,relatime shared:136 - ext4 /dev/sda1 rw
277 357 8:1 /var/snap/lxd/common/lxd/devices /var/snap/lxd/common/lxd/devices rw,relatime shared:137 master:1 - ext4 /dev/sda1 rw
404 357 0:49 / /var/snap/lxd/common/ns rw,relatime master:140 - tmpfs tmpfs rw,size=1024k,mode=700
398 268 0:49 / /var/lib/snapd/hostfs/var/snap/lxd/common/ns rw,relatime master:140 - tmpfs tmpfs rw,size=1024k,mode=700
407 357 0:50 / /var/snap/lxd/common/shmounts rw,relatime shared:143 - tmpfs tmpfs rw,size=1024k,mode=711
387 326 7:0 /wrappers/kmod /bin/kmod ro,nodev,relatime master:30 - squashfs /dev/loop0 ro
388 326 8:1 /boot /boot rw,relatime master:1 - ext4 /dev/sda1 rw
360 326 0:51 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
332 326 0:52 / /etc rw,relatime - tmpfs tmpfs rw,mode=755
361 326 8:1 /usr/share/ca-certificates /usr/share/ca-certificates rw,relatime master:1 - ext4 /dev/sda1 rw
362 268 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=61871588k,nr_inodes=15467897,mode=755
367 268 0:4 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
369 268 0:21 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
389 407 0:54 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:138 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
391 407 0:55 / /var/snap/lxd/common/shmounts/containers rw,relatime shared:139 - tmpfs tmpfs rw,size=100k,mode=711
393 357 0:56 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
411 273 0:63 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Ddu-xxx rw,relatime shared:145 - zfs lxd/containers/Ddu-xxx rw,xattr,noacl
421 273 0:73 / /var/snap/lxd/common/lxd/storage-pools/default/containers/FDT-xxx rw,relatime shared:146 - zfs lxd/containers/FDT-xxx rw,xattr,noacl
442 273 0:83 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GWR-xxx rw,relatime shared:147 - zfs lxd/containers/GWR-xxx rw,xattr,noacl
463 273 0:93 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JaN-xxx rw,relatime shared:148 - zfs lxd/containers/JaN-xxx rw,xattr,noacl
596 273 0:113 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KEK-xxx rw,relatime shared:150 - zfs lxd/containers/KEK-xxx rw,xattr,noacl
702 273 0:133 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LZT-xxx rw,relatime shared:152 - zfs lxd/containers/LZT-xxx rw,xattr,noacl
808 273 0:153 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Okf-xxx rw,relatime shared:154 - zfs lxd/containers/Okf-xxx rw,xattr,noacl
861 273 0:163 / /var/snap/lxd/common/lxd/storage-pools/default/containers/OuD-xxx rw,relatime shared:155 - zfs lxd/containers/OuD-xxx rw,xattr,noacl
914 273 0:173 / /var/snap/lxd/common/lxd/storage-pools/default/containers/SFk-xxx rw,relatime shared:156 - zfs lxd/containers/SFk-xxx rw,xattr,noacl
967 273 0:183 / /var/snap/lxd/common/lxd/storage-pools/default/containers/UuA-xxx rw,relatime shared:157 - zfs lxd/containers/UuA-xxx rw,xattr,noacl
972 273 0:193 / /var/snap/lxd/common/lxd/storage-pools/default/containers/bxN-xxx rw,relatime shared:158 - zfs lxd/containers/bxN-xxx rw,xattr,noacl
1083 273 0:223 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hqS-xxx rw,relatime shared:161 - zfs lxd/containers/hqS-xxx rw,xattr,noacl
1409 273 0:233 / /var/snap/lxd/common/lxd/storage-pools/default/containers/jbg-xxx rw,relatime shared:162 - zfs lxd/containers/jbg-xxx rw,xattr,noacl
1531 273 0:243 / /var/snap/lxd/common/lxd/storage-pools/default/containers/pej-xxx rw,relatime shared:163 - zfs lxd/containers/pej-xxx rw,xattr,noacl
1552 273 0:253 / /var/snap/lxd/common/lxd/storage-pools/default/containers/qFg-xxx rw,relatime shared:164 - zfs lxd/containers/qFg-xxx rw,xattr,noacl
1747 273 0:264 / /var/snap/lxd/common/lxd/storage-pools/default/containers/rvQ-xxx rw,relatime shared:168 - zfs lxd/containers/rvQ-xxx rw,xattr,noacl
1870 273 0:275 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tnk-xxx rw,relatime shared:172 - zfs lxd/containers/tnk-xxx rw,xattr,noacl
1971 273 0:285 / /var/snap/lxd/common/lxd/storage-pools/default/containers/udD-xxx rw,relatime shared:173 - zfs lxd/containers/udD-xxx rw,xattr,noacl
2008 273 0:295 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vEj-xxx rw,relatime shared:174 - zfs lxd/containers/vEj-xxx rw,xattr,noacl
2103 273 0:305 / /var/snap/lxd/common/lxd/storage-pools/default/containers/wmv-xxx rw,relatime shared:175 - zfs lxd/containers/wmv-xxx rw,xattr,noacl
1581 273 0:315 / /var/snap/lxd/common/lxd/storage-pools/default/containers/yUm-xxx rw,relatime shared:176 - zfs lxd/containers/yUm-xxx rw,xattr,noacl
1602 273 0:325 / /var/snap/lxd/common/lxd/storage-pools/default/containers/yXd-xxx rw,relatime shared:177 - zfs lxd/containers/yXd-xxx rw,xattr,noacl
1655 273 0:335 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zPk-xxx rw,relatime shared:178 - zfs lxd/containers/zPk-xxx rw,xattr,noacl
2501 273 0:345 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KcU-xxx rw,relatime shared:179 - zfs lxd/containers/KcU-xxx rw,xattr,noacl
2522 273 0:355 / /var/snap/lxd/common/lxd/storage-pools/default/containers/RsK-xxx rw,relatime shared:180 - zfs lxd/containers/RsK-xxx rw,xattr,noacl
2601 273 0:365 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KCU-xxx rw,relatime shared:181 - zfs lxd/containers/KCU-xxx rw,xattr,noacl
2836 273 0:375 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LCe-xxx rw,relatime shared:182 - zfs lxd/containers/LCe-xxx rw,xattr,noacl
3074 273 0:395 / /var/snap/lxd/common/lxd/storage-pools/default/containers/chQ-xxx rw,relatime shared:184 - zfs lxd/containers/chQ-xxx rw,xattr,noacl
3095 273 0:405 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dgH-xxx rw,relatime shared:185 - zfs lxd/containers/dgH-xxx rw,xattr,noacl
3216 273 0:415 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hcT-xxx rw,relatime shared:186 - zfs lxd/containers/hcT-xxx rw,xattr,noacl
3289 273 0:425 / /var/snap/lxd/common/lxd/storage-pools/default/containers/nSr-xxx rw,relatime shared:187 - zfs lxd/containers/nSr-xxx rw,xattr,noacl
3443 273 0:445 / /var/snap/lxd/common/lxd/storage-pools/default/containers/rYV-xxx rw,relatime shared:189 - zfs lxd/containers/rYV-xxx rw,xattr,noacl
3755 273 0:485 / /var/snap/lxd/common/lxd/storage-pools/default/containers/chp-xxx rw,relatime shared:193 - zfs lxd/containers/chp-xxx rw,xattr,noacl
3778 273 0:495 / /var/snap/lxd/common/lxd/storage-pools/default/containers/eKQ-xxx rw,relatime shared:194 - zfs lxd/containers/eKQ-xxx rw,xattr,noacl
4010 273 0:505 / /var/snap/lxd/common/lxd/storage-pools/default/containers/rrV-xxx rw,relatime shared:195 - zfs lxd/containers/rrV-xxx rw,xattr,noacl
4212 273 0:515 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tGV-xxx rw,relatime shared:196 - zfs lxd/containers/tGV-xxx rw,xattr,noacl
4322 273 0:525 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vKz-xxx rw,relatime shared:197 - zfs lxd/containers/vKz-xxx rw,xattr,noacl
4484 273 0:535 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Bxr-xxx rw,relatime shared:198 - zfs lxd/containers/Bxr-xxx rw,xattr,noacl
4935 273 0:575 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Jfd-xxx rw,relatime shared:202 - zfs lxd/containers/Jfd-xxx rw,xattr,noacl
4956 273 0:585 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LbB-xxx rw,relatime shared:203 - zfs lxd/containers/LbB-xxx rw,xattr,noacl
5059 273 0:605 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Rhr-xxx rw,relatime shared:205 - zfs lxd/containers/Rhr-xxx rw,xattr,noacl
5112 273 0:615 / /var/snap/lxd/common/lxd/storage-pools/default/containers/SIp-xxx rw,relatime shared:206 - zfs lxd/containers/SIp-xxx rw,xattr,noacl
5352 273 0:635 / /var/snap/lxd/common/lxd/storage-pools/default/containers/gFE-xxx rw,relatime shared:208 - zfs lxd/containers/gFE-xxx rw,xattr,noacl
5537 273 0:645 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hLx-xxx rw,relatime shared:209 - zfs lxd/containers/hLx-xxx rw,xattr,noacl
5630 273 0:665 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hid-xxx rw,relatime shared:211 - zfs lxd/containers/hid-xxx rw,xattr,noacl
6177 273 0:715 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xTo-xxx rw,relatime shared:216 - zfs lxd/containers/xTo-xxx rw,xattr,noacl
6309 273 0:725 / /var/snap/lxd/common/lxd/storage-pools/default/containers/BWK-xxx rw,relatime shared:217 - zfs lxd/containers/BWK-xxx rw,xattr,noacl
8884 273 0:1066 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dhm-xxx rw,relatime shared:248 - zfs lxd/containers/dhm-xxx rw,xattr,noacl
8932 273 0:448 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lNq-xxx rw,relatime shared:167 - zfs lxd/containers/lNq-xxx rw,xattr,noacl
9274 273 0:1141 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ejf-xxx rw,relatime shared:254 - zfs lxd/containers/ejf-xxx rw,xattr,noacl
9989 273 0:1190 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GhC-xxx rw,relatime shared:255 - zfs lxd/containers/GhC-xxx rw,xattr,noacl
10094 273 0:1201 / /var/snap/lxd/common/lxd/storage-pools/default/containers/HOp-xxx rw,relatime shared:256 - zfs lxd/containers/HOp-xxx rw,xattr,noacl
9981 273 0:1221 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cex-xxx rw,relatime shared:258 - zfs lxd/containers/cex-xxx rw,xattr,noacl
10559 273 0:1251 / /var/snap/lxd/common/lxd/storage-pools/default/containers/BJi-xxx rw,relatime shared:261 - zfs lxd/containers/BJi-xxx rw,xattr,noacl
6819 273 0:1234 / /var/snap/lxd/common/lxd/storage-pools/default/containers/AbA-xxx rw,relatime shared:165 - zfs lxd/containers/AbA-xxx rw,xattr,noacl
1498 273 0:446 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Njk-xxx rw,relatime shared:166 - zfs lxd/containers/Njk-xxx rw,xattr,noacl
5087 273 0:555 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zHm-xxx rw,relatime shared:170 - zfs lxd/containers/zHm-xxx rw,xattr,noacl
977 273 0:203 / /var/snap/lxd/common/lxd/storage-pools/default/containers/fkB-xxx rw,relatime shared:159 - zfs lxd/containers/fkB-xxx rw,xattr,noacl
3641 273 0:568 / /var/snap/lxd/common/lxd/storage-pools/default/containers/REt-xxx rw,relatime shared:171 - zfs lxd/containers/REt-xxx rw,xattr,noacl
2722 273 0:571 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IsT-xxx rw,relatime shared:188 - zfs lxd/containers/IsT-xxx rw,xattr,noacl
406 273 0:58 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zMK-staging-xxx rw,relatime shared:144 - zfs lxd/containers/zMK-staging-xxx rw,xattr,noacl
511 273 0:103 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ptR-staging-xxx rw,relatime shared:149 - zfs lxd/containers/ptR-staging-xxx rw,xattr,noacl
1089 273 0:150 / /var/snap/lxd/common/lxd/storage-pools/default/containers/nKZ-staging-xxx rw,relatime shared:153 - zfs lxd/containers/nKZ-staging-xxx rw,xattr,noacl
1371 273 0:220 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KVn-staging-xxx rw,relatime shared:160 - zfs lxd/containers/KVn-staging-xxx rw,xattr,noacl
1661 273 0:471 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xev-staging-xxx rw,relatime shared:191 - zfs lxd/containers/xev-staging-xxx rw,xattr,noacl
1817 273 0:481 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IZV-staging-xxx rw,relatime shared:192 - zfs lxd/containers/IZV-staging-xxx rw,xattr,noacl
1894 273 0:551 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Kmg-staging-xxx rw,relatime shared:199 - zfs lxd/containers/Kmg-staging-xxx rw,xattr,noacl
3598 273 0:602 / /var/snap/lxd/common/lxd/storage-pools/default/containers/nbh-staging-xxx rw,relatime shared:207 - zfs lxd/containers/nbh-staging-xxx rw,xattr,noacl
4719 273 0:712 / /var/snap/lxd/common/lxd/storage-pools/default/containers/oqz-staging-xxx rw,relatime shared:210 - zfs lxd/containers/oqz-staging-xxx rw,xattr,noacl
3903 273 0:771 / /var/snap/lxd/common/lxd/storage-pools/default/containers/kQK-staging-xxx rw,relatime shared:213 - zfs lxd/containers/kQK-staging-xxx rw,xattr,noacl
4784 273 0:781 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cHY-staging-xxx rw,relatime shared:214 - zfs lxd/containers/cHY-staging-xxx rw,xattr,noacl
4896 273 0:791 / /var/snap/lxd/common/lxd/storage-pools/default/containers/XQf-staging-xxx rw,relatime shared:215 - zfs lxd/containers/XQf-staging-xxx rw,xattr,noacl
545 273 0:130 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lmP-staging-xxx rw,relatime shared:151 - zfs lxd/containers/lmP-staging-xxx rw,xattr,noacl
1478 273 0:470 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LQO-staging-xxx rw,relatime shared:183 - zfs lxd/containers/LQO-staging-xxx rw,xattr,noacl
1392 273 0:398 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JFO-staging-xxx rw,relatime shared:169 - zfs lxd/containers/JFO-staging-xxx rw,xattr,noacl
3596 273 0:564 / /var/snap/lxd/common/lxd/storage-pools/default/containers/iNS-staging-xxx rw,relatime shared:220 - zfs lxd/containers/iNS-staging-xxx rw,xattr,noacl
4736 273 0:563 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IqL-xxx rw,relatime shared:190 - zfs lxd/containers/IqL-xxx rw,xattr,noacl
1915 273 0:469 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vxt-staging-xxx rw,relatime shared:200 - zfs lxd/containers/vxt-staging-xxx rw,xattr,noacl
1926 273 0:828 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dJH-xxx rw,relatime shared:201 - zfs lxd/containers/dJH-xxx rw,xattr,noacl
2573 273 0:703 / /var/snap/lxd/common/lxd/storage-pools/default/containers/bjb-staging-xxx rw,relatime shared:223 - zfs lxd/containers/bjb-staging-xxx rw,xattr,noacl
1573 273 0:702 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Zgw-staging-xxx rw,relatime shared:218 - zfs lxd/containers/Zgw-staging-xxx rw,xattr,noacl
6627 273 0:766 / /var/snap/lxd/common/lxd/storage-pools/default/containers/MGQ-xxx rw,relatime shared:212 - zfs lxd/containers/MGQ-xxx rw,xattr,noacl
3836 273 0:631 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Pgt-xxx rw,relatime shared:225 - zfs lxd/containers/Pgt-xxx rw,xattr,noacl
4856 274 0:838 / /var/lib/snapd/hostfs/run/user/0 rw,nosuid,nodev,relatime master:224 - tmpfs tmpfs rw,size=12376604k,mode=700

ubojan commented 4 years ago

@stgraber I've got another example here. Lemme know what should we look for.

root@server2:~# nsenter --mount=/run/snapd/ns/lxd.mnt -- cat /proc/self/mountinfo
275 373 8:1 / /var/lib/snapd/hostfs rw,relatime master:1 - ext4 /dev/sda1 rw
281 275 0:23 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=12376620k,mode=755
282 281 0:25 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
283 281 0:23 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=12376620k,mode=755
306 275 8:15 / /var/lib/snapd/hostfs/boot/efi rw,relatime master:31 - vfat /dev/sda15 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
307 275 7:0 / /var/lib/snapd/hostfs/snap/google-cloud-sdk/104 ro,nodev,relatime master:32 - squashfs /dev/loop0 ro
308 275 7:1 / /var/lib/snapd/hostfs/snap/lxd/11824 ro,nodev,relatime master:33 - squashfs /dev/loop1 ro
309 275 7:2 / /var/lib/snapd/hostfs/snap/core/7396 ro,nodev,relatime master:34 - squashfs /dev/loop2 ro
310 275 7:3 / /var/lib/snapd/hostfs/snap/lxd/12224 ro,nodev,relatime master:35 - squashfs /dev/loop3 ro
311 275 7:4 / /var/lib/snapd/hostfs/snap/google-cloud-sdk/97 ro,nodev,relatime master:36 - squashfs /dev/loop4 ro
312 275 7:5 / /var/lib/snapd/hostfs/snap/core/7917 ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
320 274 7:5 / / ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
321 320 0:6 / /dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=61869572k,nr_inodes=15467393,mode=755
322 321 0:22 / /dev/pts rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
323 321 0:24 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw
324 321 0:42 / /dev/hugepages rw,relatime master:27 - hugetlbfs hugetlbfs rw,pagesize=2M
325 321 0:19 / /dev/mqueue rw,relatime master:28 - mqueue mqueue rw
327 320 8:1 /home /home rw,relatime master:1 - ext4 /dev/sda1 rw
328 320 8:1 /root /root rw,relatime master:1 - ext4 /dev/sda1 rw
329 320 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
330 329 0:41 / /proc/sys/fs/binfmt_misc rw,relatime master:25 - autofs systemd-1 rw,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16020
331 320 0:21 / /sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
332 331 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
333 331 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:9 - tmpfs tmpfs ro,mode=755
334 333 0:27 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime master:10 - cgroup2 cgroup rw
335 333 0:28 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd
336 333 0:30 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,cpuset,clone_children
337 333 0:31 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,cpu,cpuacct
338 333 0:32 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,blkio
339 333 0:33 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,net_cls,net_prio
340 333 0:34 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,rdma
341 333 0:35 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,pids
342 333 0:36 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,perf_event
343 333 0:37 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,freezer
344 333 0:38 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,memory
345 333 0:39 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,devices
346 333 0:40 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,hugetlb
347 331 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
348 331 0:8 / /sys/kernel/debug rw,relatime master:26 - debugfs debugfs rw
349 331 0:20 / /sys/kernel/config rw,relatime master:29 - configfs configfs rw
350 331 0:43 / /sys/fs/fuse/connections rw,relatime master:30 - fusectl fusectl rw
351 320 8:1 /tmp /tmp rw,relatime master:1 - ext4 /dev/sda1 rw
352 320 8:1 /var/snap /var/snap rw,relatime master:1 - ext4 /dev/sda1 rw
353 320 8:1 /var/lib/snapd /var/lib/snapd rw,relatime master:1 - ext4 /dev/sda1 rw
354 320 8:1 /var/tmp /var/tmp rw,relatime master:1 - ext4 /dev/sda1 rw
358 320 8:1 /lib/modules /lib/modules rw,relatime master:1 - ext4 /dev/sda1 rw
359 320 8:1 /usr/src /usr/src rw,relatime master:1 - ext4 /dev/sda1 rw
360 320 8:1 /var/log /var/log rw,relatime master:1 - ext4 /dev/sda1 rw
361 320 8:1 /media /media rw,relatime shared:1 - ext4 /dev/sda1 rw
363 320 8:1 /mnt /mnt rw,relatime master:1 - ext4 /dev/sda1 rw
366 320 8:1 /snap /snap rw,relatime master:1 - ext4 /dev/sda1 rw
367 366 7:0 / /snap/google-cloud-sdk/104 ro,nodev,relatime master:32 - squashfs /dev/loop0 ro
368 366 7:1 / /snap/lxd/11824 ro,nodev,relatime master:33 - squashfs /dev/loop1 ro
369 366 7:2 / /snap/core/7396 ro,nodev,relatime master:34 - squashfs /dev/loop2 ro
370 366 7:3 / /snap/lxd/12224 ro,nodev,relatime master:35 - squashfs /dev/loop3 ro
371 366 7:4 / /snap/google-cloud-sdk/97 ro,nodev,relatime master:36 - squashfs /dev/loop4 ro
372 366 7:5 / /snap/core/7917 ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
373 353 8:1 /var/lib/snapd/hostfs /var/lib/snapd/hostfs rw,relatime - ext4 /dev/sda1 rw
276 351 8:1 /tmp/snap.lxd/tmp /tmp rw,relatime - ext4 /dev/sda1 rw
277 322 0:45 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
278 321 0:45 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
404 352 8:1 /var/snap/lxd/common/lxd/storage-pools /var/snap/lxd/common/lxd/storage-pools rw,relatime shared:239 - ext4 /dev/sda1 rw
451 352 8:1 /var/snap/lxd/common/lxd/devices /var/snap/lxd/common/lxd/devices rw,relatime shared:240 master:1 - ext4 /dev/sda1 rw
513 352 0:47 / /var/snap/lxd/common/ns rw,relatime master:245 - tmpfs tmpfs rw,size=1024k,mode=700
507 275 0:47 / /var/lib/snapd/hostfs/var/snap/lxd/common/ns rw,relatime master:245 - tmpfs tmpfs rw,size=1024k,mode=700
516 352 0:48 / /var/snap/lxd/common/shmounts rw,relatime shared:250 - tmpfs tmpfs rw,size=1024k,mode=711
492 320 7:3 /wrappers/kmod /bin/kmod ro,nodev,relatime master:35 - squashfs /dev/loop3 ro
493 320 8:1 /boot /boot rw,relatime master:1 - ext4 /dev/sda1 rw
355 320 0:49 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
326 320 0:50 / /etc rw,relatime - tmpfs tmpfs rw,mode=755
356 320 8:1 /usr/share/ca-certificates /usr/share/ca-certificates rw,relatime master:1 - ext4 /dev/sda1 rw
357 275 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=61869572k,nr_inodes=15467393,mode=755
362 275 0:4 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
364 275 0:21 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
494 516 0:52 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:241 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
496 516 0:53 / /var/snap/lxd/common/shmounts/containers rw,relatime shared:242 - tmpfs tmpfs rw,size=100k,mode=711
498 352 0:54 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
523 404 0:56 / /var/snap/lxd/common/lxd/storage-pools/default/containers/TeZ-xxx rw,relatime shared:254 - zfs lxd/containers/TeZ-xxx rw,xattr,posixacl
633 330 0:61 / /proc/sys/fs/binfmt_misc rw,relatime master:255 - binfmt_misc binfmt_misc rw
528 404 0:63 / /var/snap/lxd/common/lxd/storage-pools/default/containers/VnJ-xxx rw,relatime shared:262 - zfs lxd/containers/VnJ-xxx rw,xattr,posixacl
533 404 0:73 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Bro-xxx rw,relatime shared:263 - zfs lxd/containers/Bro-xxx rw,xattr,posixacl
559 404 0:87 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xOf-xxx rw,relatime shared:264 - zfs lxd/containers/xOf-xxx rw,xattr,posixacl
575 404 0:93 / /var/snap/lxd/common/lxd/storage-pools/default/containers/NtI-xxx rw,relatime shared:265 - zfs lxd/containers/NtI-xxx rw,xattr,posixacl
617 404 0:107 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ZlR-xxx rw,relatime shared:266 - zfs lxd/containers/ZlR-xxx rw,xattr,posixacl
725 404 0:113 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hpv-xxx rw,relatime shared:267 - zfs lxd/containers/hpv-xxx rw,xattr,posixacl
778 404 0:123 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Dlj-xxx rw,relatime shared:268 - zfs lxd/containers/Dlj-xxx rw,xattr,posixacl
831 404 0:133 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Fjj-xxx rw,relatime shared:269 - zfs lxd/containers/Fjj-xxx rw,xattr,posixacl
889 404 0:147 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GsZ-xxx rw,relatime shared:270 - zfs lxd/containers/GsZ-xxx rw,xattr,posixacl
942 404 0:157 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IRz-xxx rw,relatime shared:271 - zfs lxd/containers/IRz-xxx rw,xattr,posixacl
995 404 0:167 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Ixk-xxx rw,relatime shared:272 - zfs lxd/containers/Ixk-xxx rw,xattr,posixacl
1043 404 0:173 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JHN-xxx rw,relatime shared:273 - zfs lxd/containers/JHN-xxx rw,xattr,posixacl
1112 404 0:187 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KGc-xxx rw,relatime shared:274 - zfs lxd/containers/KGc-xxx rw,xattr,posixacl
1117 404 0:193 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KhS-xxx rw,relatime shared:275 - zfs lxd/containers/KhS-xxx rw,xattr,posixacl
1216 404 0:207 / /var/snap/lxd/common/lxd/storage-pools/default/containers/OJl-xxx rw,relatime shared:276 - zfs lxd/containers/OJl-xxx rw,xattr,posixacl
1372 404 0:217 / /var/snap/lxd/common/lxd/storage-pools/default/containers/QwN-xxx rw,relatime shared:277 - zfs lxd/containers/QwN-xxx rw,xattr,posixacl
1843 404 0:237 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Uyr-xxx rw,relatime shared:279 - zfs lxd/containers/Uyr-xxx rw,xattr,posixacl
1851 404 0:246 / /var/snap/lxd/common/lxd/storage-pools/default/containers/WCP-xxx rw,relatime shared:280 - zfs lxd/containers/WCP-xxx rw,xattr,posixacl
2133 404 0:255 / /var/snap/lxd/common/lxd/storage-pools/default/containers/XYf-xxx rw,relatime shared:281 - zfs lxd/containers/XYf-xxx rw,xattr,posixacl
2189 404 0:263 / /var/snap/lxd/common/lxd/storage-pools/default/containers/YaV-xxx rw,relatime shared:282 - zfs lxd/containers/YaV-xxx rw,xattr,posixacl
2247 404 0:277 / /var/snap/lxd/common/lxd/storage-pools/default/containers/YcJ-xxx rw,relatime shared:283 - zfs lxd/containers/YcJ-xxx rw,xattr,posixacl
2336 404 0:282 / /var/snap/lxd/common/lxd/storage-pools/default/containers/akM-xxx rw,relatime shared:284 - zfs lxd/containers/akM-xxx rw,xattr,posixacl
2430 404 0:293 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cKQ-xxx rw,relatime shared:285 - zfs lxd/containers/cKQ-xxx rw,xattr,posixacl
2464 404 0:303 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cQs-xxx rw,relatime shared:286 - zfs lxd/containers/cQs-xxx rw,xattr,posixacl
2730 404 0:313 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cux-xxx rw,relatime shared:287 - zfs lxd/containers/cux-xxx rw,xattr,posixacl
2756 404 0:327 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dIR-xxx rw,relatime shared:288 - zfs lxd/containers/dIR-xxx rw,xattr,posixacl
2773 404 0:333 / /var/snap/lxd/common/lxd/storage-pools/default/containers/eLN-xxx rw,relatime shared:289 - zfs lxd/containers/eLN-xxx rw,xattr,posixacl
2842 404 0:357 / /var/snap/lxd/common/lxd/storage-pools/default/containers/jWm-xxx rw,relatime shared:291 - zfs lxd/containers/jWm-xxx rw,xattr,posixacl
2873 404 0:367 / /var/snap/lxd/common/lxd/storage-pools/default/containers/mSe-xxx rw,relatime shared:292 - zfs lxd/containers/mSe-xxx rw,xattr,posixacl
3256 404 0:377 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tCf-xxx rw,relatime shared:293 - zfs lxd/containers/tCf-xxx rw,xattr,posixacl
3139 404 0:387 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tli-xxx rw,relatime shared:294 - zfs lxd/containers/tli-xxx rw,xattr,posixacl
3149 404 0:397 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tut-xxx rw,relatime shared:295 - zfs lxd/containers/tut-xxx rw,xattr,posixacl
3346 404 0:407 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tym-xxx rw,relatime shared:296 - zfs lxd/containers/tym-xxx rw,xattr,posixacl
3664 404 0:417 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Ezw-xxx rw,relatime shared:297 - zfs lxd/containers/Ezw-xxx rw,xattr,posixacl
3736 404 0:423 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GpS-xxx rw,relatime shared:298 - zfs lxd/containers/GpS-xxx rw,xattr,posixacl
3839 404 0:433 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IDS-xxx rw,relatime shared:299 - zfs lxd/containers/IDS-xxx rw,xattr,posixacl
4014 404 0:453 / /var/snap/lxd/common/lxd/storage-pools/default/containers/TWm-xxx rw,relatime shared:301 - zfs lxd/containers/TWm-xxx rw,xattr,posixacl
4021 404 0:463 / /var/snap/lxd/common/lxd/storage-pools/default/containers/nod-xxx rw,relatime shared:302 - zfs lxd/containers/nod-xxx rw,xattr,posixacl
4069 404 0:473 / /var/snap/lxd/common/lxd/storage-pools/default/containers/oed-xxx rw,relatime shared:303 - zfs lxd/containers/oed-xxx rw,xattr,posixacl
4090 404 0:483 / /var/snap/lxd/common/lxd/storage-pools/default/containers/sBy-xxx rw,relatime shared:304 - zfs lxd/containers/sBy-xxx rw,xattr,posixacl
4111 404 0:493 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ssW-xxx rw,relatime shared:305 - zfs lxd/containers/ssW-xxx rw,xattr,posixacl
4169 404 0:507 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ubY-xxx rw,relatime shared:306 - zfs lxd/containers/ubY-xxx rw,xattr,posixacl
4186 404 0:515 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zjy-xxx rw,relatime shared:307 - zfs lxd/containers/zjy-xxx rw,xattr,posixacl
4334 404 0:523 / /var/snap/lxd/common/lxd/storage-pools/default/containers/CIU-xxx rw,relatime shared:308 - zfs lxd/containers/CIU-xxx rw,xattr,posixacl
4614 404 0:533 / /var/snap/lxd/common/lxd/storage-pools/default/containers/CaA-xxx rw,relatime shared:309 - zfs lxd/containers/CaA-xxx rw,xattr,posixacl
4668 404 0:543 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ECh-xxx rw,relatime shared:310 - zfs lxd/containers/ECh-xxx rw,xattr,posixacl
4760 404 0:553 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Fkk-xxx rw,relatime shared:311 - zfs lxd/containers/Fkk-xxx rw,xattr,posixacl
4781 404 0:563 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GlK-xxx rw,relatime shared:312 - zfs lxd/containers/GlK-xxx rw,xattr,posixacl
410 404 0:576 / /var/snap/lxd/common/lxd/storage-pools/default/containers/HBQ-xxx rw,relatime shared:186 - zfs lxd/containers/HBQ-xxx rw,xattr,posixacl
4820 404 0:586 / /var/snap/lxd/common/lxd/storage-pools/default/containers/HdL-xxx rw,relatime shared:187 - zfs lxd/containers/HdL-xxx rw,xattr,posixacl
4826 404 0:592 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Hky-xxx rw,relatime shared:188 - zfs lxd/containers/Hky-xxx rw,xattr,posixacl
5315 404 0:606 / /var/snap/lxd/common/lxd/storage-pools/default/containers/IsS-xxx rw,relatime shared:189 - zfs lxd/containers/IsS-xxx rw,xattr,posixacl
5460 404 0:611 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Jcz-xxx rw,relatime shared:190 - zfs lxd/containers/Jcz-xxx rw,xattr,posixacl
4858 404 0:622 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JuI-xxx rw,relatime shared:191 - zfs lxd/containers/JuI-xxx rw,xattr,posixacl
5822 404 0:636 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LGl-xxx rw,relatime shared:192 - zfs lxd/containers/LGl-xxx rw,xattr,posixacl
5832 404 0:646 / /var/snap/lxd/common/lxd/storage-pools/default/containers/LXh-xxx rw,relatime shared:193 - zfs lxd/containers/LXh-xxx rw,xattr,posixacl
5789 404 0:656 / /var/snap/lxd/common/lxd/storage-pools/default/containers/MXw-xxx rw,relatime shared:194 - zfs lxd/containers/MXw-xxx rw,xattr,posixacl
5981 404 0:662 / /var/snap/lxd/common/lxd/storage-pools/default/containers/NjE-xxx rw,relatime shared:195 - zfs lxd/containers/NjE-xxx rw,xattr,posixacl
6099 404 0:676 / /var/snap/lxd/common/lxd/storage-pools/default/containers/RqF-xxx rw,relatime shared:196 - zfs lxd/containers/RqF-xxx rw,xattr,posixacl
6204 404 0:682 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Tua-xxx rw,relatime shared:197 - zfs lxd/containers/Tua-xxx rw,xattr,posixacl
6421 404 0:692 / /var/snap/lxd/common/lxd/storage-pools/default/containers/acB-xxx rw,relatime shared:198 - zfs lxd/containers/acB-xxx rw,xattr,posixacl
6462 404 0:702 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ceJ-xxx rw,relatime shared:199 - zfs lxd/containers/ceJ-xxx rw,xattr,posixacl
6042 404 0:712 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dJR-xxx rw,relatime shared:200 - zfs lxd/containers/dJR-xxx rw,xattr,posixacl
6064 404 0:722 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ejL-xxx rw,relatime shared:201 - zfs lxd/containers/ejL-xxx rw,xattr,posixacl
6499 404 0:732 / /var/snap/lxd/common/lxd/storage-pools/default/containers/fBJ-xxx rw,relatime shared:202 - zfs lxd/containers/fBJ-xxx rw,xattr,posixacl
6521 404 0:742 / /var/snap/lxd/common/lxd/storage-pools/default/containers/fqk-xxx rw,relatime shared:203 - zfs lxd/containers/fqk-xxx rw,xattr,posixacl
6621 404 0:752 / /var/snap/lxd/common/lxd/storage-pools/default/containers/fwM-xxx rw,relatime shared:204 - zfs lxd/containers/fwM-xxx rw,xattr,posixacl
6636 404 0:766 / /var/snap/lxd/common/lxd/storage-pools/default/containers/jYK-xxx rw,relatime shared:205 - zfs lxd/containers/jYK-xxx rw,xattr,posixacl
6767 404 0:772 / /var/snap/lxd/common/lxd/storage-pools/default/containers/mTx-xxx rw,relatime shared:206 - zfs lxd/containers/mTx-xxx rw,xattr,posixacl
6788 404 0:782 / /var/snap/lxd/common/lxd/storage-pools/default/containers/mav-xxx rw,relatime shared:207 - zfs lxd/containers/mav-xxx rw,xattr,posixacl
7169 404 0:792 / /var/snap/lxd/common/lxd/storage-pools/default/containers/qhO-xxx rw,relatime shared:208 - zfs lxd/containers/qhO-xxx rw,xattr,posixacl
7443 404 0:812 / /var/snap/lxd/common/lxd/storage-pools/default/containers/tWL-xxx rw,relatime shared:210 - zfs lxd/containers/tWL-xxx rw,xattr,posixacl
7464 404 0:822 / /var/snap/lxd/common/lxd/storage-pools/default/containers/uCm-xxx rw,relatime shared:211 - zfs lxd/containers/uCm-xxx rw,xattr,posixacl
7494 404 0:832 / /var/snap/lxd/common/lxd/storage-pools/default/containers/wyu-xxx rw,relatime shared:212 - zfs lxd/containers/wyu-xxx rw,xattr,posixacl
7533 404 0:842 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xFl-xxx rw,relatime shared:213 - zfs lxd/containers/xFl-xxx rw,xattr,posixacl
7574 404 0:852 / /var/snap/lxd/common/lxd/storage-pools/default/containers/yBW-xxx rw,relatime shared:214 - zfs lxd/containers/yBW-xxx rw,xattr,posixacl
7976 404 0:862 / /var/snap/lxd/common/lxd/storage-pools/default/containers/yqv-xxx rw,relatime shared:215 - zfs lxd/containers/yqv-xxx rw,xattr,posixacl
8172 404 0:873 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zGS-xxx rw,relatime shared:216 - zfs lxd/containers/zGS-xxx rw,xattr,posixacl
8314 404 0:882 / /var/snap/lxd/common/lxd/storage-pools/default/containers/AMI-xxx rw,relatime shared:217 - zfs lxd/containers/AMI-xxx rw,xattr,posixacl
8335 404 0:892 / /var/snap/lxd/common/lxd/storage-pools/default/containers/AzT-xxx rw,relatime shared:218 - zfs lxd/containers/AzT-xxx rw,xattr,posixacl
8543 404 0:908 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Dgz-xxx rw,relatime shared:219 - zfs lxd/containers/Dgz-xxx rw,xattr,posixacl
8378 404 0:916 / /var/snap/lxd/common/lxd/storage-pools/default/containers/GxO-xxx rw,relatime shared:220 - zfs lxd/containers/GxO-xxx rw,xattr,posixacl
8663 404 0:923 / /var/snap/lxd/common/lxd/storage-pools/default/containers/HPt-xxx rw,relatime shared:221 - zfs lxd/containers/HPt-xxx rw,xattr,posixacl
8852 404 0:932 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JhS-xxx rw,relatime shared:222 - zfs lxd/containers/JhS-xxx rw,xattr,posixacl
8923 404 0:942 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Jsd-xxx rw,relatime shared:223 - zfs lxd/containers/Jsd-xxx rw,xattr,posixacl
9008 404 0:952 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Kdz-xxx rw,relatime shared:224 - zfs lxd/containers/Kdz-xxx rw,xattr,posixacl
8725 404 0:962 / /var/snap/lxd/common/lxd/storage-pools/default/containers/NWD-xxx rw,relatime shared:225 - zfs lxd/containers/NWD-xxx rw,xattr,posixacl
9109 404 0:972 / /var/snap/lxd/common/lxd/storage-pools/default/containers/RNw-xxx rw,relatime shared:226 - zfs lxd/containers/RNw-xxx rw,xattr,posixacl
9015 404 0:992 / /var/snap/lxd/common/lxd/storage-pools/default/containers/aUJ-xxx rw,relatime shared:228 - zfs lxd/containers/aUJ-xxx rw,xattr,posixacl
9050 404 0:1012 / /var/snap/lxd/common/lxd/storage-pools/default/containers/dqe-xxx rw,relatime shared:230 - zfs lxd/containers/dqe-xxx rw,xattr,posixacl
9315 404 0:1022 / /var/snap/lxd/common/lxd/storage-pools/default/containers/enj-xxx rw,relatime shared:231 - zfs lxd/containers/enj-xxx rw,xattr,posixacl
9631 404 0:1031 / /var/snap/lxd/common/lxd/storage-pools/default/containers/hKk-xxx rw,relatime shared:232 - zfs lxd/containers/hKk-xxx rw,xattr,posixacl
9796 404 0:1042 / /var/snap/lxd/common/lxd/storage-pools/default/containers/iRv-xxx rw,relatime shared:233 - zfs lxd/containers/iRv-xxx rw,xattr,posixacl
9893 404 0:1052 / /var/snap/lxd/common/lxd/storage-pools/default/containers/kUl-xxx rw,relatime shared:234 - zfs lxd/containers/kUl-xxx rw,xattr,posixacl
9898 404 0:1062 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lQa-xxx rw,relatime shared:235 - zfs lxd/containers/lQa-xxx rw,xattr,posixacl
9903 404 0:1072 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lad-xxx rw,relatime shared:236 - zfs lxd/containers/lad-xxx rw,xattr,posixacl
9919 404 0:1082 / /var/snap/lxd/common/lxd/storage-pools/default/containers/pPS-xxx rw,relatime shared:237 - zfs lxd/containers/pPS-xxx rw,xattr,posixacl
9972 404 0:1092 / /var/snap/lxd/common/lxd/storage-pools/default/containers/qDe-xxx rw,relatime shared:238 - zfs lxd/containers/qDe-xxx rw,xattr,posixacl
10517 404 0:1112 / /var/snap/lxd/common/lxd/storage-pools/default/containers/qVM-xxx rw,relatime shared:246 - zfs lxd/containers/qVM-xxx rw,xattr,posixacl
10755 404 0:1122 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vCk-xxx rw,relatime shared:256 - zfs lxd/containers/vCk-xxx rw,xattr,posixacl
10577 404 0:1132 / /var/snap/lxd/common/lxd/storage-pools/default/containers/vFf-xxx rw,relatime shared:257 - zfs lxd/containers/vFf-xxx rw,xattr,posixacl
10042 404 0:1142 / /var/snap/lxd/common/lxd/storage-pools/default/containers/APJ-staging-xxx rw,relatime shared:313 - zfs lxd/containers/APJ-staging-xxx rw,xattr,posixacl
10068 404 0:1152 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Cdh-staging-xxx rw,relatime shared:314 - zfs lxd/containers/Cdh-staging-xxx rw,xattr,posixacl
10073 404 0:1162 / /var/snap/lxd/common/lxd/storage-pools/default/containers/DJB-staging-xxx rw,relatime shared:315 - zfs lxd/containers/DJB-staging-xxx rw,xattr,posixacl
10094 404 0:1172 / /var/snap/lxd/common/lxd/storage-pools/default/containers/FeI-staging-xxx rw,relatime shared:316 - zfs lxd/containers/FeI-staging-xxx rw,xattr,posixacl
10109 404 0:1182 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Goh-staging-xxx rw,relatime shared:317 - zfs lxd/containers/Goh-staging-xxx rw,xattr,posixacl
11562 404 0:1212 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JAW-staging-xxx rw,relatime shared:320 - zfs lxd/containers/JAW-staging-xxx rw,xattr,posixacl
11583 404 0:1222 / /var/snap/lxd/common/lxd/storage-pools/default/containers/JHU-staging-xxx rw,relatime shared:321 - zfs lxd/containers/JHU-staging-xxx rw,xattr,posixacl
11588 404 0:1232 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Jyc-staging-xxx rw,relatime shared:322 - zfs lxd/containers/Jyc-staging-xxx rw,xattr,posixacl
10863 404 0:1242 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KWE-staging-xxx rw,relatime shared:323 - zfs lxd/containers/KWE-staging-xxx rw,xattr,posixacl
11660 404 0:1252 / /var/snap/lxd/common/lxd/storage-pools/default/containers/KZr-staging-xxx rw,relatime shared:324 - zfs lxd/containers/KZr-staging-xxx rw,xattr,posixacl
12193 404 0:1272 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Lvj-staging-xxx rw,relatime shared:326 - zfs lxd/containers/Lvj-staging-xxx rw,xattr,posixacl
12214 404 0:1282 / /var/snap/lxd/common/lxd/storage-pools/default/containers/OlN-staging-xxx rw,relatime shared:327 - zfs lxd/containers/OlN-staging-xxx rw,xattr,posixacl
12144 404 0:1292 / /var/snap/lxd/common/lxd/storage-pools/default/containers/QJX-staging-xxx rw,relatime shared:328 - zfs lxd/containers/QJX-staging-xxx rw,xattr,posixacl
12149 404 0:1302 / /var/snap/lxd/common/lxd/storage-pools/default/containers/TAv-staging-xxx rw,relatime shared:329 - zfs lxd/containers/TAv-staging-xxx rw,xattr,posixacl
12170 404 0:1312 / /var/snap/lxd/common/lxd/storage-pools/default/containers/TSh-staging-xxx rw,relatime shared:330 - zfs lxd/containers/TSh-staging-xxx rw,xattr,posixacl
12237 404 0:1322 / /var/snap/lxd/common/lxd/storage-pools/default/containers/UHR-staging-xxx rw,relatime shared:331 - zfs lxd/containers/UHR-staging-xxx rw,xattr,posixacl
12258 404 0:1332 / /var/snap/lxd/common/lxd/storage-pools/default/containers/UWE-staging-xxx rw,relatime shared:332 - zfs lxd/containers/UWE-staging-xxx rw,xattr,posixacl
12279 404 0:1342 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Xwb-staging-xxx rw,relatime shared:333 - zfs lxd/containers/Xwb-staging-xxx rw,xattr,posixacl
12308 404 0:1352 / /var/snap/lxd/common/lxd/storage-pools/default/containers/Xyi-staging-xxx rw,relatime shared:334 - zfs lxd/containers/Xyi-staging-xxx rw,xattr,posixacl
12329 404 0:1362 / /var/snap/lxd/common/lxd/storage-pools/default/containers/bhO-staging-xxx rw,relatime shared:335 - zfs lxd/containers/bhO-staging-xxx rw,xattr,posixacl
12340 404 0:1372 / /var/snap/lxd/common/lxd/storage-pools/default/containers/daw-staging-xxx rw,relatime shared:336 - zfs lxd/containers/daw-staging-xxx rw,xattr,posixacl
12350 404 0:1382 / /var/snap/lxd/common/lxd/storage-pools/default/containers/eYU-staging-xxx rw,relatime shared:337 - zfs lxd/containers/eYU-staging-xxx rw,xattr,posixacl
12360 404 0:1392 / /var/snap/lxd/common/lxd/storage-pools/default/containers/ees-staging-xxx rw,relatime shared:338 - zfs lxd/containers/ees-staging-xxx rw,xattr,posixacl
12381 404 0:1402 / /var/snap/lxd/common/lxd/storage-pools/default/containers/flo-staging-xxx rw,relatime shared:339 - zfs lxd/containers/flo-staging-xxx rw,xattr,posixacl
12391 404 0:1412 / /var/snap/lxd/common/lxd/storage-pools/default/containers/gtS-staging-xxx rw,relatime shared:340 - zfs lxd/containers/gtS-staging-xxx rw,xattr,posixacl
12656 404 0:1432 / /var/snap/lxd/common/lxd/storage-pools/default/containers/iJi-staging-xxx rw,relatime shared:342 - zfs lxd/containers/iJi-staging-xxx rw,xattr,posixacl
12839 404 0:1442 / /var/snap/lxd/common/lxd/storage-pools/default/containers/iJv-staging-xxx rw,relatime shared:343 - zfs lxd/containers/iJv-staging-xxx rw,xattr,posixacl
12935 404 0:1452 / /var/snap/lxd/common/lxd/storage-pools/default/containers/jIJ-staging-xxx rw,relatime shared:344 - zfs lxd/containers/jIJ-staging-xxx rw,xattr,posixacl
12940 404 0:1462 / /var/snap/lxd/common/lxd/storage-pools/default/containers/kuz-staging-xxx rw,relatime shared:345 - zfs lxd/containers/kuz-staging-xxx rw,xattr,posixacl
12990 404 0:1472 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lQe-staging-xxx rw,relatime shared:346 - zfs lxd/containers/lQe-staging-xxx rw,xattr,posixacl
13043 404 0:1482 / /var/snap/lxd/common/lxd/storage-pools/default/containers/omH-staging-xxx rw,relatime shared:347 - zfs lxd/containers/omH-staging-xxx rw,xattr,posixacl
12623 404 0:1502 / /var/snap/lxd/common/lxd/storage-pools/default/containers/sqG-staging-xxx rw,relatime shared:349 - zfs lxd/containers/sqG-staging-xxx rw,xattr,posixacl
13052 404 0:1512 / /var/snap/lxd/common/lxd/storage-pools/default/containers/swe-staging-xxx rw,relatime shared:350 - zfs lxd/containers/swe-staging-xxx rw,xattr,posixacl
13062 404 0:1522 / /var/snap/lxd/common/lxd/storage-pools/default/containers/uel-staging-xxx rw,relatime shared:351 - zfs lxd/containers/uel-staging-xxx rw,xattr,posixacl
13099 404 0:1532 / /var/snap/lxd/common/lxd/storage-pools/default/containers/wnv-staging-xxx rw,relatime shared:352 - zfs lxd/containers/wnv-staging-xxx rw,xattr,posixacl
13104 404 0:1542 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xpq-staging-xxx rw,relatime shared:353 - zfs lxd/containers/xpq-staging-xxx rw,xattr,posixacl
13157 404 0:1552 / /var/snap/lxd/common/lxd/storage-pools/default/containers/yjq-staging-xxx rw,relatime shared:354 - zfs lxd/containers/yjq-staging-xxx rw,xattr,posixacl
13177 404 0:1572 / /var/snap/lxd/common/lxd/storage-pools/default/containers/zqR-staging-xxx rw,relatime shared:356 - zfs lxd/containers/zqR-staging-xxx rw,xattr,posixacl
9011 404 0:1499 / /var/snap/lxd/common/lxd/storage-pools/default/containers/xmQ-staging-xxx rw,relatime shared:243 - zfs lxd/containers/xmQ-staging-xxx rw,xattr,posixacl
9361 404 0:1586 / /var/snap/lxd/common/lxd/storage-pools/default/containers/csp-staging-xxx rw,relatime shared:251 - zfs lxd/containers/csp-staging-xxx rw,xattr,posixacl
10144 404 0:1620 / /var/snap/lxd/common/lxd/storage-pools/default/containers/lfU-xxx rw,relatime shared:253 - zfs lxd/containers/lfU-xxx rw,xattr,posixacl
9477 404 0:1613 / /var/snap/lxd/common/lxd/storage-pools/default/containers/WsX-staging-xxx rw,relatime shared:252 - zfs lxd/containers/WsX-staging-xxx rw,xattr,posixacl
10162 404 0:1262 / /var/snap/lxd/common/lxd/storage-pools/default/containers/gjx-xxx rw,relatime shared:325 - zfs lxd/containers/gjx-xxx rw,xattr,posixacl
10718 404 0:1652 / /var/snap/lxd/common/lxd/storage-pools/default/containers/cJa-xxx rw,relatime shared:358 - zfs lxd/containers/cJa-xxx rw,xattr,posixacl
3361 404 0:809 / /var/snap/lxd/common/lxd/storage-pools/default/containers/CGJ-staging-xxx rw,relatime shared:209 - zfs lxd/containers/CGJ-staging-xxx rw,xattr,posixacl
2843 404 0:348 / /var/snap/lxd/common/lxd/storage-pools/default/containers/fFU-staging-xxx rw,relatime shared:319 - zfs lxd/containers/fFU-staging-xxx rw,xattr,posixacl
2821 404 0:347 / /var/snap/lxd/common/lxd/storage-pools/default/containers/bSb-staging-xxx rw,relatime shared:229 - zfs lxd/containers/bSb-staging-xxx rw,xattr,posixacl
3078 281 0:990 / /var/lib/snapd/hostfs/run/user/0 rw,nosuid,nodev,relatime master:227 - tmpfs tmpfs rw,size=12376616k,mode=700

stgraber commented 4 years ago

Ok, the output above looks correct. Is the container listed in the log a running container?

If so, it may be a process in the container keeping it active. In which case, the reboot obviously fixed that, but so would manually killing the container, starting LXD and then starting the container back up.

ubojan commented 4 years ago

Yup, this is running container:

t=2020-02-19T16:07:08+0000 lvl=eror msg="Failed to start the daemon: Failed to run: zfs set mountpoint=/var/snap/lxd/common/lxd/storage-pools/default/containers/xxx-xxxxxxx canmount=noauto lxd/containers/xxx-xxxxxxx: umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/xxx-xxxxxxx: mountpoint not found\ncannot unmount '/var/snap/lxd/common/shmounts/storage-pools/default/containers/xxx-xxxxxxx': umount failed"

When you say killing a container, do you mean killing specific process inside container?

stgraber commented 4 years ago

No, the error suggests you need the entire dataset to be unmounted which would need the entire container to be stopped.

ubojan commented 4 years ago

OK. As lxd service is non-functional after failed update, lxc stop/start wouldn't work. So do you think something like this would work or you have better idea on mind?

TARGET_CONTAINER=xxx-xxxxxxx
cd /sys/fs/cgroup/; cat */lxc.payload/$TARGET_CONTAINER/cgroup.procs | sort -u | xargs kill

stgraber commented 4 years ago

You can send kill -37 to the systemd (init) process of the container which should cause a clean setup. If you don't care about that, sending SIGKILL to the [lxc monitor] process will kill the entire container.

ubojan commented 4 years ago

Sorry for delay... I've tested it out some more and have some feedback.

Your suggestion works just fine. Thanks! I was able to "fix" lxd service without server reboot.

Only negative side effect of it is that I would have to do the procedure on several containers in the row (when I kill one, another issue appears in lxd log; I kill that one, third one appears in log, etc), so I have to kill all of those (like 2% or 3% of total containers on host) before lxd starts to run properly and I am able to start killed containers.

I've found out a solution to killing containers: instead of killing them, this seems to work fine: zfs set mountpoint=/var/snap/lxd/common/lxd/storage-pools/default/containers/xxx-xxxxxx canmount=noauto lxd/containers/xxx-xxxxxx It works fine, no server or container downtime at all and lxd service gets "fixed".

I've noticed one issue which seems to be pretty serious - no matter if lxd snap update goes fine or I have to fix it: snapshot creation fails on random containers (more on servers where I have to fix lxd than the rest of them): Create instance snapshot (mount source): Failed to run: zfs mount lxd/containers/xxx-xxxxxx: cannot mount 'lxd/containers/xxx-xxxxxx': filesystem already mounted That can be resolved with zfs mount, but it takes a lot of time (cause issues are apparently random). Is there a way to check for these "mount issues" before they happen, right after lxd update? Otherwise I won't be able to avoid actually rebooting each server where I do lxd update :(

stgraber commented 4 years ago

Hmm, so you may need to make sure things are consistent on your system.

Normally what we'd expect is:

All containers datasets to have a mountpoint= property set
All containers datasets to have canmount=noauto set
Only running containers to have their dataset mounted

root@vorash:~# nsenter --mount=/run/snapd/ns/lxd.mnt env LD_LIBRARY_PATH=/snap/lxd/current/zfs-0.8/lib/ /snap/lxd/current/zfs-0.8/bin/zfs list -r internal/lxd/containers -o mountpoint,canmount,mounted
MOUNTPOINT                                                                  CANMOUNT  MOUNTED
none                                                                              on       no
/var/snap/lxd/common/lxd/storage-pools/internal/containers/bcfg2s01           noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/candid01           noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/dnses01            noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/dnsr03             noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/dnsr05             noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/dnssec01           noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/freeradius01       noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/git01              noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/jenkins01          noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/lxc-discourse01    noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/lxc-kpi01          noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/mailman01          noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/nextcloud01        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/nsec01             noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/piferrari01        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/rbac01             noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/rproxy01           noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/rsyslog01          noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/samba-dc01         noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/samba-web01        noauto       no
/var/snap/lxd/common/lxd/storage-pools/internal/containers/shell01            noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/smtpout01          noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/unifi01            noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/vpn01              noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/web-dl01           noauto      yes
/var/snap/lxd/common/lxd/storage-pools/internal/containers/wordpress01        noauto      yes

In this example on one of my servers, only the samba-web01 container is stopped, everything else is running an all options are set appropriately (pool is named internal).

ubojan commented 4 years ago

When I run provided command on server where snapshot creation fails at least on one container (lQC-xxx in this case), everything seems to be in order (stopped containers are not mounted):

MOUNTPOINT                                                                                       CANMOUNT  MOUNTED
none                                                                                                   on       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/BOs-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/BRH-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/BYu-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/CIE-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/CQm-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/DBG-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/DBU-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Drk-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Eay-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/FSn-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Gij-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/HBj-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/HPF-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Hha-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Hxr-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/HzT-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/IzF-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/JDL-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/JIe-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/JLY-staging-xxx                        noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/JcR-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Jjc-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/JkS-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/JsK-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/KIQ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/KRL-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Kqp-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/LLb-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Lsb-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Mpo-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/MwV-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/OGx-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Pes-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Pge-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Pjw-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/PsW-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/QSa-xxx                                noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/Qbd-xxx                                noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/Qhv-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/QsL-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/ROR-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/RUL-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/SmS-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Snj-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/TJC-xxx                                noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/TjE-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/TmD-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/TpN-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/Uqv-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/UvZ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/UxI-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/VDL-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/VRS-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/VZy-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/VsC-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/WMl-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/WUU-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/XIb-xxx                                noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/YEt-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/YFr-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/YQU-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/YWN-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/acr-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/avp-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/awE-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/bTX-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/bhd-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/bjy-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/dLv-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/edW-xxx                                noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/ejX-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/etK-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/gLU-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/gOH-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/grX-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/hYP-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/hjQ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jAZ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jJO-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jNM-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jal-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jbj-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/jdC-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/xxx                                    noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/klk-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/lQC-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/lbN-staging-xxx                        noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/lvx-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/mJF-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/mir-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/mrE-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/nVU-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/nZG-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/oAs-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/pbs-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/pft-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/poJ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/qPT-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/qjz-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/rFt-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/rrL-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/rzo-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/sAF-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/sBU-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/sWt-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/sgN-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/slN-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/tiE-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/uzE-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/vFj-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/vkA-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/wDP-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/wGL-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/wJu-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/wTR-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/waZ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/xGJ-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/xPr-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/xjV-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/yWC-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/ygi-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/yiP-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/ykW-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/ylz-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/ysf-staging-xxx                        noauto       no
/var/snap/lxd/common/lxd/storage-pools/default/containers/zTb-xxx                                noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/zbw-staging-xxx                        noauto      yes
/var/snap/lxd/common/lxd/storage-pools/default/containers/zjN-xxx                                noauto      yes

But still snapshot creation on lQC-xxx fails. Then I've ran: zfs mount lxd/containers/lQC-xxx to fix the issue.

When I run command again, relevant line is same:

/var/snap/lxd/common/lxd/storage-pools/default/containers/lQC-xxx                              noauto      yes

Line is same also when I run command after server reboot:

/var/snap/lxd/common/lxd/storage-pools/default/containers/lQC-xxx                              noauto      yes

So it looks like provided command can't be used to detect snapshot creation issue in this case.

stgraber commented 4 years ago

@ubojan that smells like some kind of zfs bug then... Is running zfs mount for every one of your running containers as feasible workaround for this (rather than rebooting)?

ubojan commented 4 years ago

I've tried that today and it seems to be working so far. I'll do more thorough tests tomorrow.

ubojan commented 4 years ago

It looks like that does the trick. Thanks !

canonical / lxd

Lxd service can't start after 3.18 -> 3.21 update #6901

Required information

Issue description

Steps to reproduce

Information to attach