How to connect two clusters, networking and kubectl?

[tetricky]

Please let me know if this is inappropriate for here, and I'll close and research independently.

I'm struggling with my usage case, which I believe is possible with kilo.

I have a k3s cluster (with kilo) in kvm virtual machines using NAT. These are on a host with a public ip (local network can ping to cluster). I want to use the cluster to run backend services. (longhorn, postgresql etc). I then also want to run a single node cluster on the host machine, to connect to the backend service cluster and provide public facing services.

I want to be able to kubectl to both clusters (I like octant to monitor/manipulate cluster status). I also want to share services between clusters (predominantly backend to single node cluster). I am currently using kubectl from the host to access the backend cluster. I want to be able to do this remotely as I could for a cluster with a public ip.

It seems that the correct approach might be to setup kilo vpn between the two clusters. Then vpn server to provide a public endpoint on the public facing single node instance.

My head is going a bit fluffy with all the internconnected parts, and what needs to be where. (order to connect the two clusters, applying metallb to clusters, how to identify and access the independent clusters and kubectl to them).

I'm assuming that I will need to install kgctl on the kvm host.

Any guidance will be appreciated.

[squat]

Hi @tetricky this is definitely the right place to ask :))

It sounds like what you're trying to do is to create multi-cluster services, i.e. run a Kubernetes Service in one cluster and make use of it from a Pod running in another cluster. This is totally possible with Kilo and in fact I run this setup all of the time :))

There's a doc in the Kilo repo that explains how to do this: https://github.com/squat/kilo/blob/master/docs/multi-cluster-services.md

Please take a look at it and see if it answers your questions. Note: the scripts in the doc assume you have kgctl installed on your local machine, e.g. the laptop from which you run kubectl, but kgctl is not required on any of the cluster machines.

[tetricky]

Is that complicated if I can't reach the cluster behind NAT from my local machine? I will try running it from the kvm host (which can reach the NAT).

Do I need Kilo running on both clusters (the VM based cluster, plus the host)? I note that running kilo on both clusters starts wg with the same CIDR, do I need to change that?

I find that I am very confused by the what, where, and in what order.

Then how to use kilo to access kubectl on both clusters, from remote - presumably a wireguard interface on my workstation vpn'd to the cluster(s)?

I'm working from https://kilo.squat.ai/docs/multi-cluster-services/ is $SERVICECIDR1 and $SERVICECIDR2 the kilo, or the cluster CIDR? (I seem to have 10.4.0.0 for both the wireguard subnets, and I have defined 10.42.0.0/16 and 10.10.0.0/16 for my k3s cluster CIDR's).

Apologies if this is very stupid stuff, I've got some basic knowledge, but only with the sort of things I've done before. This is new to me.

[squat]

Hi @tetricky, that's a lot of questions; I'll try my best to answer them all:

• yes, Kilo will need to be installed on both clusters;
• as the doc mentions, the Kilo CIDR must be different on the clusters: e.g. one can be 10.4/16 and the other 10.5/16;
• the Service CIDRs for the clusters (10.43/16 by default on k3s) must be different if you are connecting from one cluster to another cluster's Service IP
• the Pod CIDRs for the clusters (10.42/16 by default on k3s) must be different if you are connecting from one cluster to another cluster's Pod IP
• SERVICECIDR refers to the cluster's Service CIDR

[tetricky]

I accept that I might be missing something, but I genuinely can't find this:

"as the doc mentions, the Kilo CIDR must be different on the clusters: e.g. one can be 10.4/16 and the other 10.5/16;"

I've looked at the manifests. I do not know how to set it differently for different clusters.

[squat]

Hi @tetricky the doc mentions this in the second paragraph:

https://github.com/squat/kilo/blob/72f5107979281076a32fb1639d135db6bf89ac80/docs/multi-cluster-services.md#L8

Please let me know if as a user this isn't clear enough and maybe how it could be improved! I'm always very happy for any help and contributions :)

Regarding modifying the Kilo CIDR, the Kilo binary that runs on all your nodes has a --subnet flag that can be used to modify the WireGuard CIDR: https://github.com/squat/kilo/blob/master/cmd/kg/main.go#L97 These command line options definitely need a reference document. I'll work on this now :)

[tetricky]

Perhaps out of scope here, but in terms of k3s, I'm now passing a number of flags to setup the appropriate environment;

export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik --no-deploy flannel --tls-san --cluster-cidr 10.10.0.0/16 --service-cidr 10.11.0.0/16"

So that leaves me with two clusters. Service CIDR 10.43.0.0/16; Pod CIDR 10.42.0.0/16 (vm_cluster) Service CIDR 10.11.0.0/16; Pod CIDR 10.10.0.0/16 (kvm_host)

I've edited kilo-k3s.yaml to add --mesh-granularity=full --subnet=10.5.0.0/16 (I had to experiment a bit with how it was expected)

Then I set: export KUBECONFIG1=[path_to_kvm_host_kubeconfig] export KUBECONFIG2=[path_to_vm_cluster_kubeconfig]

Cutting and pasting the kgctl script from the multi-cluster services page, gives me:

$ for n in $(kubectl --kubeconfig $KUBECONFIG1 get no -o name | cut -d'/' -f2); do

# Specify the service CIDR as an extra IP range that should be routable.
kgctl --kubeconfig $KUBECONFIG1 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR1 | kubectl --kubeconfig $KUBECONFIG2 apply -f -

done Error: flag needs an argument: --allowed-ips Usage: kgctl showconf node [name] [flags]

so what I'm taking from that is that I need to be specifiying somehow an "allowed ip"

What is this, how do I identify and specify it?

...or am I doing something else very stupid wrongly?

Thank you for your help. I'm edging forward. I really appreciate that your help on this is beyond reasonable.

----------------Edit------------- Right. I switched users, to use my "go" environment, so of course the CIDR variables weren't set in that environment. Setting and trying again.

[tetricky]

So now....after the first script I get output of:

peer.kilo.squat.ai/[kvm_hostname] created

...and after the second:

peer.kilo.squat.ai/[vm_cluster_master_hostname] created

Error: failed to initialize peer backend: failed to create CRD: Post "https://192.168.122.101:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 192.168.122.101:6443: i/o timeout

failed to initialize peer backend: failed to create CRD: Post "https://192.168.122.101:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 192.168.122.101:6443: i/o timeout error: no objects passed to apply E0918 11:20:03.922681 4978 reflector.go:126] github.com/squat/kilo/pkg/k8s/backend.go:147: Failed to list *v1.Node: Get "https://192.168.122.101:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.122.101:6443: i/o timeout

It then retries periodically, and times out every time. The ip address is the virtual machine cluster master ip address (NAT subnet).

kgctl graph

digraph kilo { label="10.10.0.0/24"; labelloc=t; outputorder=nodesfirst; overlap=false; "[kvm_host_hostname]"->"[vm_cluster_hostname]"[ dir=both, style=dashed ]; subgraph "clusterlocation" { label=""; style="dashed,rounded"; "[kvm_host_hostname]" [ label=" [kvm_host_hostname] 10.10.0.0/24 10.10.0.0 10.5.0.1 [public_ip]:51820", rank=1, shape=ellipse ];

} ; subgraph "cluster_peers" { label="peers"; style="dashed,rounded"; "[vm_cluster_hostname]" [ label="[vm_cluster_hostname]

", shape=ellipse ];

} ;

}

[squat]

Ack, the clusters looks good 👍 Yes the problem in the first set of logs was that the env variables for SERVICECIDRX were not set.

Just to be sure, everything directly above is the output to the terminal from running the second snippet of registering the VMs with the kvm host?

Did any of the VMs successfully register? Maybe only the first VM and the rest are causing issues in kgctl?

This seems like something caused the kvm API server to become unusable. Can you still use kubectl against it? E.g. kubectl get ns?

[tetricky]

If I export KUBECONFIG=$KUBECONFIG1 kubectl get all --all-namespaces

(kvm host) then I get the expected output.

export KUBECONFIG=$KUBECONFIG2 kubectl get all --all-namespaces

Unable to connect to the server: dial tcp 192.168.122.101:6443: i/o timeout

It hangs, no output.

So you are correct. This is a bare k3s still, so the only thing running is kilo.

(Note: I was appending the kgctl graph output to the previous post, while you were replying).

Looks like I'm now unable to connect to the vm's. Might have to rebuild all that.

[tetricky]

on the kvm host, I'm getting this:

interface: kilo0 public key: ElTy0A7yA3b4B6dm8zbdNgzdVmRfVQ7e2x8Y7cbTL3Y= private key: (hidden) listening port: 51820

peer: mSLjhLugVc/VqmJBmP75RhG4wRXOPyAZ2fXq2xaOS0U= endpoint: 192.168.122.101:51820 allowed ips: 10.42.0.0/24, 192.168.122.101/32, 10.42.1.0/24, 192.168.122.102/32, 10.42.2.0/24, 192.168.122.103/32, 10.4.0.1/32, 10.43.0.0/16 transfer: 0 B received, 5.20 KiB sent

Could it be that kilo is now routing traffic over the wireguaard connection....which isn't working? (for example, I can't seem to ssh into the vm's any more).

Suspect this is not recoverable, but I could do with understanding how to uninstall cleanly from the host so that when I rebuild I'm not storing up legacy problems.

[squat]

Yes, there is definitely an issue with the fact that the private IPs of the VMs (192.168.122.101, 192.168.122.102, 192.168.122.103) are also the same as the "public" IP, i.e. 192.168.122.101. Note that the generated config uses 192.168.122.101 as the IP for the endpoint, but ALSO tries to route 192.168.122.101 via WireGuard. This will not work.

Question for you: are the private IPs for the VMs (192.168.122.101, 192.168.122.102, 192.168.122.103) actually reachable from the KVM host?

If not, then we cannot specify the endpoint here; we'll need the VMs to connect to the KVM host via WireGuard and for the endpoint to be added automatically and for the connection to be kept open via WireGuard's persistent keepalive.

If yes, then the private IPs do not need to be specified in the allowed IPs section of the configuration, since we don't need to route connections to those addresses over the VPN for connectivity.

The thing that we are more interested in is why the VM cluster's API became inaccessible. The best thing to do determine the cause would be to recreate the cluster (sorry :/) and then, rather than generating the Peer configurations and applying them directly using the snippet in the doc, simply generate them and share them here so that we can inspect further. In other words:

# Perform a dry-run of registering the nodes in cluster1 as peers of cluster2.
for n in $(kubectl --kubeconfig $KUBECONFIG1 get no -o name | cut -d'/' -f2); do
    # Specify the service CIDR as an extra IP range that should be routable.
    # Save the generated configurations to disk rather than applying it directly to the cluster.
    kgctl --kubeconfig $KUBECONFIG1 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR1 > $n.yaml
done

[tetricky]

"Question for you: are the private IPs for the VMs (192.168.122.101, 192.168.122.102, 192.168.122.103) actually reachable from the KVM host?"

No. I can see them running in libvirt, but I have no viable route to them.

From the kgctl graph above. The section: [kvm_host_hostname] 10.10.0.0/24 10.10.0.0 10.5.0.1 [public_ip]:51820", rank=1, shape=ellipse ];

The [public_ip] there, is the actual public ip (ie not the host NAT bridge). This might be wrong, as traffic has to go outside NAT, to come back in? I've added it as a firewall route to allow port 51820 udp. No observable change.

I can rebuild. How do I strip all the settings off the vm host, so I am getting back to a clean sheet of paper?

f I tear down kilo, the kilo0 wireguard interface remains. It doesn't respond to the normal wiregaurd tools (wg-quick). I need to be sure I'm getting rid of that, or I'm going to have to rebuild the kvm host, which will be time consuming.

[tetricky]

On teardown...running the k3s uninstall script, and rebooting the vm's and the kvm host, put's me back to the clean server and vm's...once more accessible from a networking perspective.

So, my plan is this.

1. Re-build the vm cluster in NAT, with the public ip as the kvm host NAT bridge.
2. Rebuild the (single node) kvm host cluster with the public ip the resolvable public ip of the host.

Then use the test code, rather than the script in the docs?

[squat]

yes exactly and please share the generated YAML files here so we can inspect :)

[tetricky]

Okay. I think I've done that, and it's created yaml file named after the host. cat hetzds01-deb.yaml apiVersion: kilo.squat.ai/v1alpha1 kind: Peer metadata: creationTimestamp: null name: hetzds01-deb spec: allowedIPs:

• 10.10.0.0/24
• 10.10.0.0/32
• 10.5.0.1/32
• 10.11.0.0/16 endpoint: ip: [public ip of host] port: 51820 presharedKey: "" publicKey: [key_removed]

[tetricky]

I've just worked out that I need to run this modified to address the other cluster as well.

apiVersion: kilo.squat.ai/v1alpha1 kind: Peer metadata: creationTimestamp: null name: debserv-k3s-101 spec: allowedIPs:

• 10.12.0.0/24
• 192.168.122.101/32
• 10.12.1.0/24
• 192.168.122.102/32
• 10.12.2.0/24
• 192.168.122.103/32
• 10.6.0.1/32
• 10.13.0.0/16 endpoint: ip: 192.168.122.101 port: 51820 presharedKey: "" publicKey: [different_key_removed]

It seems to me that the vm cluster is having the master ip as the endpoint (NAT address), and the host cluster has an endpoint of the public facing ip address, so there is no shared network or route.

I don't know how to build/configure it so that it can work in this use case - one cluster public facing, then a NAT'd cluster behind a bridge on the same host.

[tetricky]

So..looking at those, I think I need the subnet to be the subnet of the NAT (of which the host is bridge). I have changed that, and it looks better to me...I'm concerned that "192.168.122.1/32" is not the right mask...but in the kilo manifests I defined the "-- subnet" as 192.168.122.0/24.

apiVersion: kilo.squat.ai/v1alpha1 kind: Peer metadata: creationTimestamp: null name: hetzds01-deb spec: allowedIPs:

• 10.10.0.0/24
• 10.10.0.0/32
• 192.168.122.1/32
• 10.11.0.0/16 endpoint: ip: [public ip of host] port: 51820 presharedKey: "" publicKey: [key_removed]

[tetricky]

I've run the multi-cluster script. I get the first completing (so creating the config successfully, but not able to connect?), the second hangs. If I look at the wg interface I see this on the cluster (nothing changes on the kvm host). I think the issue is around the public ip address...but I can't get my tiny little mind around it.

interface: kilo0 public key: [key1_removed] private key: (hidden) listening port: 51820

peer: [key2_removed] endpoint: 192.168.122.102:51820 allowed ips: 10.12.1.0/24, 192.168.122.102/32, 10.6.0.2/32

peer: [key3_removed] endpoint: 192.168.122.103:51820 allowed ips: 10.12.2.0/24, 192.168.122.103/32, 10.6.0.3/32

peer: [key4_removed] endpoint: [public ip of host]:51820 allowed ips: 10.10.0.0/24, 10.10.0.0/32, 192.168.122.1/32, 10.11.0.0/16 transfer: 0 B received, 13.01 KiB sent

[squat]

Hmm I think there must be a problem with the topology and initial connectivity of the clusters, even before Kilo is installed. However, I am a little bit confused about the it. Can you provide some more details about where the clusters are physically running and what connectivity exists between them? If this is tricky to explain, maybe we can do some live debugging together. You can find me on the Kubernetes Slack as @squat.

[squat]

BTW, as you commented above in https://github.com/squat/kilo/issues/77#issuecomment-694533386, it was not clear how to override the configuration for the Kilo agent that runs on the nodes, e.g. to change the Kilo subnet. To improve this pain point, I added a new doc for kg in https://github.com/squat/kilo/commit/0cc1a2ff8c2605c158b42346db36c64ef8ab2dc3. This doc can be read in the repo or on the website at: https://kilo.squat.ai/docs/kg.

[tetricky]

The more I find out, the less I know. I've run into some problems understanding the hetzner networking setup, and I'm re-working that. I will come back to this when I'm convinced I have a working system on the conventional networking side.

Should it be possible to connect clusters together using kilo with ipv6, and also have a cluster with a ipv4 endpoint? I'm completely confused about what is possible, and what isn't. Never mind trying to do it.

[AntonOfTheWoods]

@tetricky how are things progressing? I am embarking on a similar journey for some Contabo VPSes :-). My goal is to have two kubernetes clusters - one in their US DC and one in Germany - and to have everything talk over wireguard. I am also struggling to understand how to set up multi-cluster communication, and I'm sure we're not alone. There seem to be quite a few el cheapo VPS providers (Hetzner, Contabo, SSDNodes, etc.) out there that would really benefit from having solid documentation on setting up a few different kinds of topology. If you have any experience I'd love to compare!

[tetricky]

I'm currently trying to put together a complete framework, for what I want to end up with. I'm learning to use gopass and summon for secret encryption, and cloud-config to deploy k3os server and agent instances.

First stage is hetzner kvm, with cluster of three k3os running backend (longhorn, databases, minio), with the server instance having an additional IP.

Then k3s on the host. Then running workloads on contabo vps's.

I'm documenting as i go along, and happy to publish or collaborate.

Its slow because I have other work that i need to prioritise. I'm working on getting all the other bits right and hoping a better structured platform will be more predictable and easier to tear down and modify and re-apply.

I'm converging on the kilo bit, but not there yet.

[AntonOfTheWoods]

Great to hear @tetricky ! I am going to take a shot at a kilo helm chart. I've always found a properly documented helm chart can make things much easier to understand. I'll publish a link when I have something to show.

[AntonOfTheWoods]

As promised, https://github.com/AntonOfTheWoods/kilo-chart. I have only implemented k3s support for the moment, but it would probably take less than an hour to implement the others cleanly. There are a few advanced options that might not work but it appears to do what the raw manifests do, and it should now be very easy to enrich with more functionality.

@squat , is a helm chart something you might be interested in having around? Do you know helm very well? If so, please have a look and tell me what you think!

[tetricky]

I gave up trying to get kilo working with a cluster split between kvm vm's and other nodes. I'm now trying to use kilo over flannel without vm's.