Storage does not creates directory tree in all cluster members

[norbertoisaac]

Required information

required-info.txt

Issue description

Storage does not creates directory tree in all cluster members

Steps to reproduce

root@fdo-kvmc1h1:~# lxc storage create --target fdo-kvmc1h1 pool_btrfs btrfs source=/storage/btrfs_pool1/subVolLxd
Storage pool pool_btrfs pending on member fdo-kvmc1h1
root@fdo-kvmc1h1:~# lxc storage create --target fdo-kvmc1h4 pool_btrfs btrfs source=/storage/btrfs_pool1/subVolLxd
Storage pool pool_btrfs pending on member fdo-kvmc1h4
root@fdo-kvmc1h1:~# lxc storage create --target fdo-kvmc1h5 pool_btrfs btrfs source=/storage/btrfs_pool1/subVolLxd
Storage pool pool_btrfs pending on member fdo-kvmc1h5
root@fdo-kvmc1h1:~# lxc storage create pool_btrfs btrfs
Storage pool pool_btrfs created
root@fdo-kvmc1h1:~# ls -l /storage/btrfs_pool1/subVolLxd
total 0
drwx--x--x 1 root root 0 Jun  3 13:48 buckets
drwx--x--x 1 root root 0 Jun  3 13:48 containers
drwx--x--x 1 root root 0 Jun  3 13:48 containers-snapshots
drwx--x--x 1 root root 0 Jun  3 13:48 custom
drwx--x--x 1 root root 0 Jun  3 13:48 custom-snapshots
drwx--x--x 1 root root 0 Jun  3 13:48 images
drwx--x--x 1 root root 0 Jun  3 13:48 virtual-machines
drwx--x--x 1 root root 0 Jun  3 13:48 virtual-machines-snapshots

root@fdo-kvmc1h4:~# btrfs subvolume list /storage/btrfs_pool1
ID 256 gen 11 top level 5 path subVolLxd
root@fdo-kvmc1h4:~# ls -l /storage/btrfs_pool1/subVolLxd
total 0
root@fdo-kvmc1h4:~# lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0                       7:0    0  38.7M  1 loop /snap/snapd/21465
loop1                       7:1    0  74.2M  1 loop /snap/core22/1380
loop2                       7:2    0 103.4M  1 loop /snap/lxd/28460
loop3                       7:3    0  38.8M  1 loop /snap/snapd/21759
loop4                       7:4    0  83.4M  1 loop /snap/microceph/981
sda                         8:0    0 278.5G  0 disk
├─sda1                      8:1    0     1M  0 part
├─sda2                      8:2    0     2G  0 part /boot
└─sda3                      8:3    0 276.5G  0 part
  ├─ubuntu--vg-ubuntu--lv 252:0    0   100G  0 lvm  /
  └─ubuntu--vg-BTRFS      252:1    0   100G  0 lvm  /storage/btrfs_pool1

root@fdo-kvmc1h5:~# btrfs subvolume list /storage/btrfs_pool1
ID 256 gen 11 top level 5 path subVolLxd
root@fdo-kvmc1h5:~# ls -l /storage/btrfs_pool1/subVolLxd/
total 0
root@fdo-kvmc1h5:~# lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0                       7:0    0  38.7M  1 loop /snap/snapd/21465
loop1                       7:1    0  74.2M  1 loop /snap/core22/1380
loop2                       7:2    0 103.4M  1 loop /snap/lxd/28460
loop3                       7:3    0  38.8M  1 loop /snap/snapd/21759
loop4                       7:4    0  83.4M  1 loop /snap/microceph/981
sda                         8:0    0 278.5G  0 disk
├─sda1                      8:1    0     1M  0 part
├─sda2                      8:2    0     2G  0 part /boot
└─sda3                      8:3    0 276.5G  0 part
  ├─ubuntu--vg-ubuntu--lv 252:0    0   100G  0 lvm  /
  └─ubuntu--vg-BTRFS      252:1    0   100G  0 lvm  /storage/btrfs_pool1
root@fdo-kvmc1h5:~# touch /storage/btrfs_pool1/subVolLxd/one
root@fdo-kvmc1h5:~# touch /storage/btrfs_pool1/subVolLxd/two
root@fdo-kvmc1h5:~# ls -l /storage/btrfs_pool1/subVolLxd/
total 0
-rw-r--r-- 1 root root 0 Jun  3 13:57 one
-rw-r--r-- 1 root root 0 Jun  3 13:57 two

Information to attach

lxc monitor > lxc-monitor.log lxc-monitor.log

[tomponline]

Yes that seems suspect to me. Does it cause any knock on issues though?

[norbertoisaac]

Yes, it causes failed instances creation on members fdo-kvmc1h4 and fdo-kvmc1h5 for that storage pool. However, later i deleted the pool_btrfs and try with success from member fdo-kvmc1h5, then all members was have the same correct folder subtree

[tomponline]

Thanks.

Do you know the reproducer steps to get LXD to create the directories only on a single host?

[norbertoisaac]

Hi The steps was the commands listed on "Steps to reproduce" of the first comment on the host fdo-kvmc1h1. After success from host fdo-kvmc1h5, i have not try again with a new storage pool creation from h1. ¿Do you need i try again a btrfs storage pool creation from h1?, if yes, tellme what kind of info migh collect for help in case of fail.