AntonOfTheWoods
commented
3 years ago Here is what I came up with to execute on bootstrap (and with the addition of every new node) for the peers:
#!/bin/bash
set -e
# TODO, think about making functions so DRY
export KUBECONFIG1=~/.kube/config.eu
export SERVICECIDR1=10.143.0.0/16
export KUBECONFIG2=~/.kube/config.am
export SERVICECIDR2=10.43.0.0/16
# Prepare all nodes as individual mesh nodes
echo "Preparing nodes for the EU cluster"
for n in $(kubectl --kubeconfig $KUBECONFIG1 get nodes -o name | cut -d'/' -f2); do
kubectl --kubeconfig $KUBECONFIG1 annotate --overwrite nodes $n kilo.squat.ai/location=$n
kubectl --kubeconfig $KUBECONFIG1 annotate --overwrite nodes $n kilo.squat.ai/force-internal-ip=192.0.2.${n#"eu"}/32
done
echo "Preparing nodes for the AM cluster"
for n in $(kubectl --kubeconfig $KUBECONFIG2 get nodes -o name | cut -d'/' -f2); do
kubectl --kubeconfig $KUBECONFIG2 annotate --overwrite nodes $n kilo.squat.ai/location=$n
kubectl --kubeconfig $KUBECONFIG2 annotate --overwrite nodes $n kilo.squat.ai/force-internal-ip=192.0.2.${n#"am"}/32
done
# Register the nodes in cluster1 as peers of cluster2.
echo "Registering EU nodes as peers of the AM nodes"
for n in $(kubectl --kubeconfig $KUBECONFIG1 get nodes -o name | cut -d'/' -f2); do
# Specify the service CIDR as an extra IP range that should be routable.
kgctl --kubeconfig $KUBECONFIG1 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR1 | kubectl --kubeconfig $KUBECONFIG2 apply -f -
done
# Register the nodes in cluster2 as peers of cluster1.
echo "Registering AM nodes as peers of the EU nodes"
for n in $(kubectl --kubeconfig $KUBECONFIG2 get nodes -o name | cut -d'/' -f2); do
# Specify the service CIDR as an extra IP range that should be routable.
kgctl --kubeconfig $KUBECONFIG2 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR2 | kubectl --kubeconfig $KUBECONFIG1 apply -f -
doneI have nodes amX and euX, and it seems to work pretty well. Does this look reasonable?
squat
Hi, this one also took several hours of my life! It appears that when you are in full mesh mode, the "leader" doesn't want to do routing for the other nodes in the cluster (or it's a bug?). The problem is that when you use
kgctlto get the peers config to send to the other cluster, it complains saying "XXX isn't a leader" for all nodes in a cluster that aren't "the leader". Annotating all the nodes as "leaders" had no effect - there is obviously some other consideration. Now the result is that all nodes in cluster A have wireguard config for only one node in cluster B (the leader), and rather than the non-leader nodes in cluster B having a route via the leader for the IPs belonging to cluster A, they have full peers.The solution that appears to work was to keep things in full mesh AND tag every node with their own node name. This allows
kgctlto export the peer config for each node properly, meaning each node in each cluster gets a peer for every other node (so full mesh between all nodes in all clusters).Is there a better way? If this is "the way", I'll add setting this value as part of the daemonset in my helm chart.