What happened?

Going on like 40h straight of no sleep as this was my weekend passion project. I'll edit this issue and post good information on a second pass. Dropping in some notes for now.

BTW .... WOW!! Thank you for this amazing Open Source contribution! Netmaker is phenomenal!

Clouds used in my weekend implementation:

Azure
Vultr
Digital Ocean

System Requirements + Costs:

Master nodes 2 vCPU + 4GB RAM (I made 5 of these - generally $30/month each)
Agent nodes 1 vCPU 1GB RAM (I made 12 of these - generally $7/month each)
Bandwith: If over the pre-allocation, generally $0.01 per GB

Goal:

Build a edge cloud for our MSP business as an "edge-cloud command center" with support for Anycast IPs and geo-redundancy with our ingress
Lay a foundation with Cilium's clustermesh to extend to worker node k8s clusters running in our on-premise datacenters and IaaS clouds (Hetzner, Vultr Bare Metal, Cherry Servers, etc.)

Networking

Made a network for the physical node layer called "snyderedge-p" as 10.254.0.0/15
Made a network that will be used in the future across K8S services/pods called "snyderedge-k" as 10.252.0.0/15
Enabled UDP hole punching and renamed by default interface

Ubuntu 22.04 node setup

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo mount bpffs -t bpf /sys/fs/bpf
echo "bpffs                  /sys/fs/bpf             bpf     defaults 0 0" | sudo tee -a /etc/fstab

Join each node to Netmaker

curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc
curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list
sudo apt update
sudo apt install netclient
sudo netclient join -t <TOKEN>

Initialize the first K3s server node with cluster-init

export SNYDEREDGE_K3S_TOKEN=mySuperSecretToken
export SNYDEREDGE_IP=$(ip a | grep snyderedge-p | grep '10.254' | awk '{print $2}' | cut -f1 -d '/')
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.26.1+k3s1 K3S_KUBECONFIG_MODE="644" sh -s - server --token=$SNYDEREDGE_K3S_TOKEN \
  --flannel-backend=none --disable=traefik --disable-network-policy \
  --cluster-domain="snyderedge.local" --cluster-cidr="10.252.0.0/16" --service-cidr="10.253.0.0/16" --cluster-dns="10.253.0.10" \
  --bind-address=$SNYDEREDGE_IP --node-ip=$SNYDEREDGE_IP --node-external-ip=$SNYDEREDGE_IP \
  --cluster-init

Add the other K3S server nodes with the control plane

export SNYDEREDGE_K3S_TOKEN=mySuperSecretToken
export SNYDEREDGE_CLUSTER_MASTER=<HOSTNAME_OF_K3S_CLUSTER_INIT_NODE>.snyderedge-p
export SNYDEREDGE_IP=$(ip a | grep snyderedge-p | grep '10.254' | awk '{print $2}' | cut -f1 -d '/')
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.26.1+k3s1 sh -s - server --token=$SNYDEREDGE_K3S_TOKEN \
  --flannel-backend=none --disable=traefik --disable-network-policy \
  --cluster-domain="snyderedge.local" --cluster-cidr="10.252.0.0/16" --service-cidr="10.253.0.0/16" --cluster-dns="10.253.0.10" \
  --bind-address=$SNYDEREDGE_IP --node-ip=$SNYDEREDGE_IP --node-external-ip=$SNYDEREDGE_IP \
  --server https://$SNYDEREDGE_CLUSTER_MASTER:6443

Add K3S agent nodes

export SNYDEREDGE_K3S_TOKEN=mySuperSecretToken
export SNYDEREDGE_CLUSTER_MASTER=<HOSTNAME_OF_K3S_CLUSTER_INIT_NODE>.snyderedge-p
export SNYDEREDGE_IP=$(ip a | grep snyderedge-p | grep '10.254' | awk '{print $2}' | cut -f1 -d '/')
curl -sfL https://get.k3s.io | K3S_URL='https://${SNYDEREDGE_CLUSTER_MASTER}:6443' sh -s - agent --token=$SNYDEREDGE_K3S_TOKEN \
  --node-ip=$SNYDEREDGE_IP --node-external-ip=$SNYDEREDGE_IP \
  --server https://$SNYDEREDGE_CLUSTER_MASTER:6443

Install Cilium

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
cilium install --version=1.13.0 --helm-auto-gen-values cilium-helm-values-install.yaml \
  --helm-set ipam.mode=cluster-pool \
  --helm-set ipam.operator.clusterPoolIPv4PodCIDRList="10.252.0.0/15" \
  --helm-set tunnel="geneve" \
  --helm-set ipv4NativeRoutingCIDR="10.252.0.0/15" \
  --helm-set bgpControlPlane.enabled=true \
  --helm-set k8s.requireIPv4PodCIDR=true \
  --helm-set device="snyderedge-p" \
  --helm-set cluster.name="snyderedge" \
  --helm-set cluster.id=1 \
  --helm-set tag="v1.13.0" \
  --helm-set kubeProxyReplacement="strict" \
  --helm-set bgpControlPlane.enabled=true \
  --helm-set k8s.requireIPv4PodCIDR=true \
  --helm-set clustermesh.useAPIServer=true \
  --helm-set clustermesh.apiserver.enabled=true \
  --helm-set clustermesh.apiserver.tls.auto.method=certmanager \
  --helm-set hubble.relay.enable=true
  #--helm-set hubble.ui.enabled=true

# Apply the Cilium configuration
cilium install --version=1.13.0 --helm-values cilium-helm-values-install.yaml

# Enable mesh
cilium clustermesh enable --service-type=LoadBalancer

Upgrade example

helm repo add cilium https://helm.cilium.io/
helm template cilium/cilium --version=1.13.0 \
  --set sctp.enabled=true \
  --namespace kube-system \
  > cilium-helm-values-upgrade.yaml
kubectl apply -f cilium-helm-values-upgrade.yaml

Cleanup One-Liners

# If needing to blow out the installation due to trial & error:
/usr/local/bin/k3s-uninstall.sh && sudo ip link delete cilium_host && sudo ip link delete cilium_vxlan

# If needing to disconnect the netmaker client
sudo netclient leave -n snyderedge-p

I'll update this issue over the coming few weeks with more rich information. Hopefully this can then make it into docs more docs for others to benefit from :

Version

v0.17.1

What OS are you using?

Linux

Relevant log output

No response

Contributing guidelines

[X] Yes, I did.

gravitl / netmaker

[Docs]: Building a cross-cloud wide edge network with K3S + Cilium CNI + Istio #2054