Error: 500 Server Error during cluster join

[VariableDeclared]

Hello when bootstrapping maas-anvil my final node fails to add:

maas-anvil cluster join     --role database --role region --role agent --role haproxy     --token TOKEN
An unexpected error has occurred. Please run 'maas-anvil inspect' to generate an inspection report.
Error: 500 Server Error: Internal Server Error for url: http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/cluster/control

Inspection report can be found here: https://drive.google.com/file/d/1RJF5NtLhNNdM9pUeV24X9Qsdk6BG3z8g/view?usp=sharing

Currently investigating why this might have happened and will update this issue once/if I find something

Peter

[VariableDeclared]

Interestingly, trying to remove the node, and start again - I appear to have some inconsistent state:

maas-anvil cluster add --name k8s-hm-2.demos.vlan32.maas
Node already a member of the MAAS cluster
ubuntu@k8s-hm-3:~$ maas-anvil cluster remove --name k8s-hm-2.demos.vlan32.maas
Error: Node does not exist in the sunbeam cluster

[VariableDeclared]

more information:

maas-anvil cluster list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ Node                       ┃ Status ┃ Region ┃ Agent ┃ Database ┃ HAProxy ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ k8s-hm-1.demos.vlan32.maas │   up   │   x    │   x   │    x     │    x    │
│ k8s-hm-2.demos.vlan32.maas │   up   │        │       │          │         │
│ k8s-hm-3.demos.vlan32.maas │   up   │   x    │   x   │    x     │    x    │
└────────────────────────────┴────────┴────────┴───────┴──────────┴─────────┘
ubuntu@k8s-hm-3:~$ maas-anvil cluster remove --name k8s-hm-2.demos.vlan32.maas
Error: Node does not exist in the sunbeam cluster

[VariableDeclared]

I could not reproduce this issue on my environment - I suspect the issue might have been time related, as such I will close this issue until I have a reason to re-open it.