Unable to join LXD Cluster on 6.1

canonical / lxd

Powerful system container and virtual machine manager

https://canonical.com/lxd

GNU Affero General Public License v3.0

4.35k stars 931 forks source link

Unable to join LXD Cluster on 6.1 #13793

Closed TorbenMartin closed 2 months ago

TorbenMartin commented 2 months ago

Hey.

Running Ubunu Server 24.04 on HOST System with LXD 6.1 Created 2 VMs with Ubuntu 24.04 and installed LXD 6.1 and want to join Cluster for VXLAN Tests behind OpenWRT LXC

lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none

But already Problems on Beginning... with joining the Cluster after fresh Install:

snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd
reboot

Server 1:

root@lxd1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.223]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=lxd1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd1:~# 
root@lxd1:~# lxc cluster add lxd2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member lxd2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjQyN2VjYTUzMDRkMTNlOTQ0Mjc3OGJmNmM2NWRkZTU5MmU3MGEwY2M0ZjdlMjQyOTQwZTY5OTFkNjgxNTUxYzIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjhlZmVjNjEyNWVlMzhmNDg3ZDhlMTBlNWE2NzhlOTExNmNkMjMyYWE1YzZkMzQ0MzA4YWM1YTA3MTAwMzJlYTQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQxMTozMzoyNC45OTYyOTgwNVoifQ==
root@lxd1:~#

Server 2:

root@lxd2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.224]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjQyN2VjYTUzMDRkMTNlOTQ0Mjc3OGJmNmM2NWRkZTU5MmU3MGEwY2M0ZjdlMjQyOTQwZTY5OTFkNjgxNTUxYzIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjhlZmVjNjEyNWVlMzhmNDg3ZDhlMTBlNWE2NzhlOTExNmNkMjMyYWE1YzZkMzQ0MzA4YWM1YTA3MTAwMzJlYTQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQxMTozMzoyNC45OTYyOTgwNVoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: lxd2
  enabled: true
  member_config:
  - entity: storage-pool
    name: default
    key: size
    value: ""
    description: '"size" property for storage pool "default"'
  - entity: storage-pool
    name: default
    key: source
    value: ""
    description: '"source" property for storage pool "default"'
  - entity: storage-pool
    name: default
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "default"'
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.0.3.223:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3jCCAWSgAwIBAgIRANF+sSzifc3aNEBnZV4bwg0wCgYIKoZIzj0EAwMwIjEM
    MAoGA1UEChMDTFhEMRIwEAYDVQQDDAlyb290QGx4ZDEwHhcNMjQwNzIyMDgyOTMw
    WhcNMzQwNzIwMDgyOTMwWjAiMQwwCgYDVQQKEwNMWEQxEjAQBgNVBAMMCXJvb3RA
    bHhkMTB2MBAGByqGSM49AgEGBSuBBAAiA2IABCgImyl74WbTyZYZnvD9nGZHiy9R
    xwIkvKg7GIHs+lYTP3OmCsDG5rlx9dfHgcnhWM/Ucf8GpAMwdS/rkuF6wrKKfHkU
    zeXl6PqHyWX3STNUuCwn50XEBRtRVcobAthjVaNeMFwwDgYDVR0PAQH/BAQDAgWg
    MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwJwYDVR0RBCAwHoIE
    bHhkMYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNoADBlAjEA
    +m/vsUhONh8CpDsVoHfIX6kJOxz/0NmyK0uECRuNDrYQwaVEn3OEMusgEkOOsG8Q
    AjBnH0jH5VjE6eS4qOPNSb8z2Je9BpTfWoZn923dpEbfHpm5APmRpgeqz87KWdu3
    R8o=
    -----END CERTIFICATE-----
  server_address: 10.0.3.224:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific

TorbenMartin commented 2 months ago

One Try again.... Cluster join works with minimal Initialization.

root@lxd1:~# lxd init --minimal
root@lxd1:~# lxd init --auto

(finaly this both work)

root@lxd1:~# lxc config set core.https_address 10.0.3.223:8443
root@lxd1:~# lxc cluster enable lxd1
Clustering enabled
root@lxd1:~# lxc cluster add lxd2
Member lxd2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjdjMTlkOTYyNGVmZmJkZWQ0ZTRlNTYyNmZkZmM3NTJjMTZiNTI0NmRmZjE5MjU2MTYyZDc3OTExNjAyZDBmYjYiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjdhYzg4Mjc0ODM4M2YyZmE1ZGJjNTkxZDUwOWU4M2Y0YTNiMDE3NjQ2NTIxM2U1OWFiNDA1NGRkMDdkZDQzNTkiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQyMToxODoyNi4wMzM1NDAxMzRaIn0=

root@lxd2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.224]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjdjMTlkOTYyNGVmZmJkZWQ0ZTRlNTYyNmZkZmM3NTJjMTZiNTI0NmRmZjE5MjU2MTYyZDc3OTExNjAyZDBmYjYiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjdhYzg4Mjc0ODM4M2YyZmE1ZGJjNTkxZDUwOWU4M2Y0YTNiMDE3NjQ2NTIxM2U1OWFiNDA1NGRkMDdkZDQzNTkiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQyMToxODoyNi4wMzM1NDAxMzRaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd2:~#

root@lxd2:~# lxc cluster list
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+

TorbenMartin commented 2 months ago

Another Error. Why this works on Single Servers but not on a Cluster

root@lxd1:~# lxc cluster list
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
root@lxd1:~# lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none
Error: Network not pending on any node (use --target <node> first)

TorbenMartin commented 2 months ago

joining lxd3 not work.

root@lxd3:~# 
snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd
snap "lxd" is not installed
2024-07-22T21:01:20Z INFO Waiting for automatic snapd restart...
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
         installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
         details.

lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
General refreshes of "lxd" held indefinitely
Restarted.
root@lxd3:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.225]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDMiLCJmaW5nZXJwcmludCI6IjRlY2JkZTBlYjE0YTVhMGZiYmU4NWYwM2UzYTAyMjY3N2U2Y2IwYjA0NDljMmU2MjU3YTJiNzcyZTNkZDNiOWIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIiwiMTAuMC4zLjIyNDo4NDQzIl0sInNlY3JldCI6IjhkYWQxZWZkMTVkNjE2Y2RkYWU3MzE5YWUxN2Q2NzNiOGMwYzU0NzA0MDkxMjc1ZTM4NTRmOGRlMWFlZTU2NjQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yM1QwMDowMzowMS45NTg4MzI0NTdaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific
root@lxd3:~#

boltmark commented 2 months ago

Thanks @TorbenMartin, I am able to reproduce this with 6.1/stable and Ubuntu 24.04 VMs. We are looking into this.

boltmark commented 2 months ago

Another Error. Why this works on Single Servers but not on a Cluster

root@lxd1:~# lxc cluster list
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
root@lxd1:~# lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none
Error: Network not pending on any node (use --target <node> first)

When defining networks in a cluster, you first need to use lxc network create with --target <node-name> for each of your nodes, while passing in any node-specific config. Once you have done this for each of the nodes, you can use the non-targeted command to create the network for the cluster.

For your reference: https://documentation.ubuntu.com/lxd/en/latest/howto/cluster_config_networks/.

TorbenMartin commented 2 months ago

I was waiting one night. Set lxc config set core.https_address 0.0.0.0:8443

And now can join Cluster with LXD3

root@lxd3:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.225]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDMiLCJmaW5nZXJwcmludCI6IjRlY2JkZTBlYjE0YTVhMGZiYmU4NWYwM2UzYTAyMjY3N2U2Y2IwYjA0NDljMmU2MjU3YTJiNzcyZTNkZDNiOWIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIiwiMTAuMC4zLjIyNDo4NDQzIl0sInNlY3JldCI6IjkwNjc3NzY4ZDFiYzliZjUwY2UyYmU4ZDgzZGJlYWM5M2U4YTJlOWRlMGZjZTliYWFkZTA0MzRiYjZmMzljZGIiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yM1QwODoxOToyMS4wMDQ2NTYzMTlaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd3:~# lxc cluster list
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database        |              |                |             |        |                   |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd3 | https://10.0.3.225:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+

tomponline commented 2 months ago

Please can you provide output of sudo snap list on each server. Thanks

tomponline commented 2 months ago

I've not been able to reproduce this issue with:

lxd 6.1-c14927a 29551 6.1/stable canonical✓ -

host 1

root@vc1:~# snap install lxd --channel=6.1/stable
2024-07-23T15:15:21Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vc1:~# lxd init 
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.21.203.10]: 
Are you joining an existing cluster? (yes/no) [default=no]:  
What member name should be used to identify this server in the cluster? [default=vc1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vc1:~# lxc cluster add vc2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vc2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZjMiIsImZpbmdlcnByaW50IjoiOGUwNmI4Mzk4YzI0YWQ5MzVhMzdjYjE5MGVkM2E2OTVhMWVlOTRhM2ZiMzcwY2M5MjI0YWRiOTA0ZGFmMTA0NyIsImFkZHJlc3NlcyI6WyIxMC4yMS4yMDMuMTA6ODQ0MyJdLCJzZWNyZXQiOiJhNjM4OWM0ZjJkM2VhMjliYWJmZTZlNmM3OTc0MzI5YzMzZjJiNzI5MTE4OTZmNTU4YWNkMjIwZmI4MTIyNjkxIiwiZXhwaXJlc19hdCI6IjIwMjQtMDctMjNUMTg6MTY6MzYuNDc3NjAwNzZaIn0=

host 2

root@vc2:~# snap install lxd --channel=6.1/stable
2024-07-23T15:17:00Z INFO Waiting for automatic snapd restart...
2024-07-23T15:17:01Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vc2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.21.203.5]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZjMiIsImZpbmdlcnByaW50IjoiOGUwNmI4Mzk4YzI0YWQ5MzVhMzdjYjE5MGVkM2E2OTVhMWVlOTRhM2ZiMzcwY2M5MjI0YWRiOTA0ZGFmMTA0NyIsImFkZHJlc3NlcyI6WyIxMC4yMS4yMDMuMTA6ODQ0MyJdLCJzZWNyZXQiOiJhNjM4OWM0ZjJkM2VhMjliYWJmZTZlNmM3OTc0MzI5YzMzZjJiNzI5MTE4OTZmNTU4YWNkMjIwZmI4MTIyNjkxIiwiZXhwaXJlc19hdCI6IjIwMjQtMDctMjNUMTg6MTY6MzYuNDc3NjAwNzZaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vc2
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.21.203.10:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB2zCCAWGgAwIBAgIRAMpQKSjwQY1LaGUEFEihzEgwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZjMTAeFw0yNDA3MjMxNTE1MTJa
    Fw0zNDA3MjExNTE1MTJaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    YzEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATuijHtSMuYa2oSNK8gpd1pjajoo3eJ
    IkhtsXAgVTmZnm/s6muC7JR7D96yjKnU6I9cpImDJ1v79isElghdO911Nzn6nre6
    NbSeqjdRNrIJtPdfVQnOc4w/mzdsAnSyigqjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zj
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNoADBlAjBOsMO/
    HyQHw4zqwm8xXdSicRAWnJsybKjCDeiFDtyFxBiJuorHBYhLANiRJz2cPy8CMQC7
    BF/e6GnS/zTOCc7dHZmnkwfHDEj49Zd5cgiVIFI2ooKOctuHJw801Fer5ra6DcI=
    -----END CERTIFICATE-----
  server_address: 10.21.203.5:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

tomponline commented 2 months ago

@boltmark do you have a reproducer?

TorbenMartin commented 2 months ago

sudo snap lis

root@lxd1:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd

[root@lxd2:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd
root@lxd2:~# ]

root@lxd3:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd

tomponline commented 2 months ago

@TorbenMartin and its working now?

boltmark commented 2 months ago

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

tomponline commented 2 months ago

@boltmark please can you post your reproducer steps in full

TorbenMartin commented 2 months ago

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

1) the process is remove lxd first then install 6.1 so there is no old data from another init... on a fresh install on 2 new VMs with 24.04

2) i not initialize lxd many times. but when initialize lxd with all the default options...-> it will create "default" and "local" storage with both zfs driver. so now for another test i run init but i choose:

Do you want to configure a new local storage pool? (yes/no) [default=yes]: no

----->

root@lxd1v2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.228]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=lxd1v2]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: no
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd1v2:~# lxc cluster add lxd2v2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

root@lxd1v2:~# lxc cluster add lxd2v2
Member lxd2v2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2MiIsImZpbmdlcnByaW50IjoiMmU2ZWIzNjc5YzY0NGNkYTJkNDEyYWNjZjk0YzBkNmJmNjdiMmE0MzY2NTgzOTExN2NjYjVlZDAwMGVjNzMwMiIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjI4Ojg0NDMiXSwic2VjcmV0IjoiYjA3N2Y5YWVkNjg4ZmU5ODRkZjMzZGNkMjZlYmJkNjMzNzU0MDdlMjY1NDU2OGNmYzNjNThjYTUyZTE0YzU4NCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjAwOjQ2LjcwNDU5OTIwM1oifQ==
root@lxd1v2:~# lxc storage list
+---------+--------+-------------+---------+---------+
|  NAME   | DRIVER | DESCRIPTION | USED BY |  STATE  |
+---------+--------+-------------+---------+---------+
| default | zfs    |             | 1       | CREATED |
+---------+--------+-------------+---------+---------+
root@lxd1v2:~#

Here LXD2v2:

root@lxd2v2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.230]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2MiIsImZpbmdlcnByaW50IjoiMmU2ZWIzNjc5YzY0NGNkYTJkNDEyYWNjZjk0YzBkNmJmNjdiMmE0MzY2NTgzOTExN2NjYjVlZDAwMGVjNzMwMiIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjI4Ojg0NDMiXSwic2VjcmV0IjoiYjA3N2Y5YWVkNjg4ZmU5ODRkZjMzZGNkMjZlYmJkNjMzNzU0MDdlMjY1NDU2OGNmYzNjNThjYTUyZTE0YzU4NCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjAwOjQ2LjcwNDU5OTIwM1oifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Choose "size" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific
root@lxd2v2:~#

TorbenMartin commented 2 months ago

@TorbenMartin and its working now?

it was working add server2 to server1 in cluster when i init the first Node with --minimal it was not working when i add server3 first. after change the core.https_address may i can add server3

this test VMs are a fresh auto/cloud-init installation by the Host.

Host + VMs have same Kernel.

root@lxd2v2:~# uname -r
6.8.0-36-generic

root@bitmaster:/home/serveradmin# uname -r
6.8.0-36-generic

The VMs are connected into a Network that i created in LXD on the Host. This Network is attached to a pfsense Firewall VM.

I just upgraded the Host a few days ago from LXD 5.21 to 6.1

tomponline commented 2 months ago

Im still quite confused what the reproducer steps are fot a fresh system. What are you doing differently compared to my working example above?

tomponline commented 2 months ago

Its sounding like you have some existing state.

TorbenMartin commented 2 months ago

Its sounding like you have some existing state. no i not run another init before this.

Did you run this Ubuntu 24.04 VMs on a LXD Hypervisor or do you run it on another Virtualization Platform?

Maybe something is wrong with the cloud-init / cloud disk image?!

Just tested on Ubuntu 22.04 VMs. Run LXD 6.1 Same Problem.

boltmark commented 2 months ago

@boltmark please can you post your reproducer steps in full

Below is the only way that I can reproduce a similar error, but this seems different from what @TorbenMartin is running into as he is not using lxd init multiple times.

Host 1:

root@vm1:~# snap install lxd --channel=6.1/stable
2024-07-23T16:57:04Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.173]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=vm1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm1:~# lxc cluster add vm2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vm2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9

Host 2:

root@vm2:~# snap install lxd --channel=6.1/stable
2024-07-23T17:21:58Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, dir, lvm, powerflex, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm2:~# lxc storage create local zfs
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Storage pool local created

root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.128]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vm3
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.252.72.173:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3DCCAWGgAwIBAgIRANF5Oq8sSdMLrAG0IN3fHCYwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZtMTAeFw0yNDA3MjMxNjU3MzRa
    Fw0zNDA3MjExNjU3MzRaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    bTEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARe1Igyywf00FAxI6FRUMSbm2+oVMHz
    fOSW6EjdhhUCrCSoagvXZ94Ig8cbpayO33UtD1+H3UJVSQGVXfrVAAlFLBYDuT09
    VcqTOts26oHLbD68C92T67I9RBdGh61u0JOjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zt
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNpADBmAjEAxUrp
    cGKk4X/Hw1yvrWA0fk7VKFT76NOfUKj1xhbP5pGqqUgg6D0OzGF3oMvc0sFNAjEA
    5D9cyYsNt0LRstiTRFXD/E7IixaAzjR+GXG+B+oo003tK49qCn6J9GlsNdjXhwXk
    -----END CERTIFICATE-----
  server_address: 10.252.72.128:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "local": Config key "size" is cluster member specific

tomponline commented 2 months ago

Its sounding like you have some existing state. no i not run another init before this.

Did you run this Ubuntu 24.04 VMs on a LXD Hypervisor or do you run it on another Virtualization Platform?

Maybe something is wrong with the cloud-init / cloud disk image?!

Just tested on Ubuntu 22.04 VMs. Run LXD 6.1 Same Problem.

I ran it inside 2 fresh ubuntu:24.04 lxd vms

TorbenMartin commented 2 months ago

I think i found out.

lxd1v4

root@lxd1v4:~# snap install lxd --channel=6.1/stable
root@lxd1v4:~# lxd init
root@lxd1v4:~# lxc cluster add lxd2v4

lxd2v4

root@lxd2v4:~# snap install lxd --channel=6.1/stable
root@lxd2v4:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.234]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NCIsImZpbmdlcnByaW50IjoiZjU0MzM0MmMxMjBkNjY1N2Q2NDQxYzk2MGJkMzJhY2M1MjlhMGRjYjE0MjJlNzBiM2UyYzJjZjYwNjVlMjEzMSIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjMzOjg0NDMiXSwic2VjcmV0IjoiZDE0OTk2MGQ5Y2IyYjU4OGVhMTg1YmRhN2M3YjZmMGE5NWZhYjA5YzJlODA5ZWQ5MzM0MzFkYjU1OGZiOGVhMiIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjQyOjM1Ljk0MjU5OTAzNFoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]

my first Process was.

apt-get update
snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd

Its sounding like you have some existing state.

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

....

maybe this not work. will "snap set lxd ui.enable=true" init LXD before i run "lxd init"?

tomponline commented 2 months ago

Ah this might be the issue:

snap restart --reload lxd

There is s bug in snapd that incorrectly initializes lxd on reload.

Try doing:

sudo systemctl reload snap.lxd.daemon instead

tomponline commented 2 months ago

@boltmark please can you post your reproducer steps in full

Below is the only way that I can reproduce a similar error, but this seems different from what @TorbenMartin is running into as he is not using lxd init multiple times.

Host 1:

root@vm1:~# snap install lxd --channel=6.1/stable
2024-07-23T16:57:04Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.173]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=vm1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm1:~# lxc cluster add vm2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vm2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9

Host 2:

root@vm2:~# snap install lxd --channel=6.1/stable
2024-07-23T17:21:58Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, dir, lvm, powerflex, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm2:~# lxc storage create local zfs
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Storage pool local created

root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.128]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vm3
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.252.72.173:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3DCCAWGgAwIBAgIRANF5Oq8sSdMLrAG0IN3fHCYwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZtMTAeFw0yNDA3MjMxNjU3MzRa
    Fw0zNDA3MjExNjU3MzRaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    bTEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARe1Igyywf00FAxI6FRUMSbm2+oVMHz
    fOSW6EjdhhUCrCSoagvXZ94Ig8cbpayO33UtD1+H3UJVSQGVXfrVAAlFLBYDuT09
    VcqTOts26oHLbD68C92T67I9RBdGh61u0JOjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zt
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNpADBmAjEAxUrp
    cGKk4X/Hw1yvrWA0fk7VKFT76NOfUKj1xhbP5pGqqUgg6D0OzGF3oMvc0sFNAjEA
    5D9cyYsNt0LRstiTRFXD/E7IixaAzjR+GXG+B+oo003tK49qCn6J9GlsNdjXhwXk
    -----END CERTIFICATE-----
  server_address: 10.252.72.128:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "local": Config key "size" is cluster member specific

Ah ok so ur initializing both hosts first and then trying to cluster them. This is quite sn unusual scenario, but lxd should be doing some consistency checks iirc to check they are compatible.

In this case it looks like the two local storage pool settings are conflicting.

TorbenMartin commented 2 months ago

Ah this might be the issue:

snap restart --reload lxd

There is s bug in snapd that incorrectly initializes lxd on reload.

Try doing:

sudo systemctl reload snap.lxd.daemon instead

My Process now: snap install lxd --channel=6.1/stable snap set lxd ui.enable=true snap refresh --hold lxd systemctl reload snap.lxd.daemon

Result: snap.lxd.daemon.service is not active, cannot reload. So may when it is not active... may this is the Problem.

root@lxd1v5:~# systemctl restart snap.lxd.daemon
root@lxd1v5:~# systemctl status snap.lxd.daemon
● snap.lxd.daemon.service - Service for snap application lxd.daemon
     Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static)
     Active: active (running) since Tue 2024-07-23 18:33:18 UTC; 5s ago
TriggeredBy: ● snap.lxd.daemon.unix.socket
   Main PID: 2410 (daemon.start)
      Tasks: 0 (limit: 4643)
     Memory: 37.0M (peak: 56.1M)
        CPU: 1.070s
     CGroup: /system.slice/snap.lxd.daemon.service
             ‣ 2410 /bin/sh /snap/lxd/29551/commands/daemon.start

Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_stat
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_swaps
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_uptime
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_slabinfo
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - shared_pidns
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - cpuview_daemon
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - loadavg_daemon
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - pidfds
Jul 23 18:33:20 lxd1v5 lxd.daemon[2410]: => Starting LXD
Jul 23 18:33:21 lxd1v5 lxd.daemon[2566]: time="2024-07-23T18:33:21Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network prio>
lines 1-21/21 (END)

Finaly i can join the Cluster. I accepted all default Options on "lxd init"

root@lxd1v5:~# lxc cluster add lxd2v5
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member lxd2v5 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NSIsImZpbmdlcnByaW50IjoiZGQ3M2M4ZTZhNjUyMzRjNGNmNjk2NTRlMmIyYzRhN2YzNmIyZGNiN2NlNWZlZWM5OWU0MmExYTA2NGU5ZjEyYyIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjM1Ojg0NDMiXSwic2VjcmV0IjoiYTdkYzZjNGFjNWIwMjFlMWUyM2EzN2RiZDRhMGQ5ZTZkMjZmMzNlNTdmNTU4MzU1ZTY1MjM4MjVjMTNmMGYyOCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIxOjM0OjQ1LjYzOTI0MDM5NFoifQ==

Join success:

root@lxd2v5:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.236]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NSIsImZpbmdlcnByaW50IjoiZGQ3M2M4ZTZhNjUyMzRjNGNmNjk2NTRlMmIyYzRhN2YzNmIyZGNiN2NlNWZlZWM5OWU0MmExYTA2NGU5ZjEyYyIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjM1Ojg0NDMiXSwic2VjcmV0IjoiYTdkYzZjNGFjNWIwMjFlMWUyM2EzN2RiZDRhMGQ5ZTZkMjZmMzNlNTdmNTU4MzU1ZTY1MjM4MjVjMTNmMGYyOCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIxOjM0OjQ1LjYzOTI0MDM5NFoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

tomponline commented 2 months ago

snap.lxd.daemon.service is not active, cannot reload. So may when it is not active... may this is the Problem.

If you've not started lxd you dont need to reload it.

tomponline commented 2 months ago

Finaly i can join the Cluster.

Ok great so the main issue is that "snap restart lxd" causes lxd to be initialized if not already which then reveals the issue @boltmark has identified.

TorbenMartin commented 2 months ago

snap.lxd.daemon.service is not active, cannot reload. So may when it is not active... may this is the Problem.

If you've not started lxd you dont need to reload it.

https://ubuntu.com/blog/lxd_ui

i didnt know that it need to be active started or initialize by myself before i enable the UI

tomponline commented 2 months ago

It doesn't since 5.21 onwards

tomponline commented 2 months ago

The snap restart issue is here btw https://bugs.launchpad.net/snapd/+bug/2028141