search

squat / kilo

Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes (k8s + wg = kg)
https://kilo.squat.ai
Apache License 2.0
2.02k stars 122 forks source link

Support running kg as a peer #16

Open SerialVelocity opened 5 years ago

SerialVelocity commented 5 years ago

Hi Squat!

It would be great if it was possible to run kg as a peer. Basically, I want a process watching kubernetes for updates and auto-updating my wireguard interface with the new routes.

I've hacked it into the codebase here: https://github.com/squat/kilo/compare/master...SerialVelocity:kg-peer but it isn't complete or very pretty code!

squat commented 5 years ago

Hm I quite like this idea. This would mean that any time you want a non-k8s-node to join a cluster, the only thing you have to do is run kg, rather than make the interfaces and register routes etc yourself.

I think the one missing piece from your PoC is that the Kubernetes nodes need to be able to discover your local host somehow. To do this, we could have the kg on the host to create a Peer to reflect itself. We’ll also need a way for this peer to reserve an address in the kilo subnet, or otherwise specify its own address in a different subnet. This would need to be an additional flag.

SerialVelocity commented 5 years ago

Yes, I added the peer manually for now and just selected the first IP (I didn't even change the /32 subnet!)

squat commented 5 years ago

Let me know if you’d like to PR it :) I’d be happy to iterate in it afterwards

SerialVelocity commented 5 years ago

Hm I quite like this idea. This would mean that any time you want a non-k8s-node to join a cluster, the only thing you have to do is run kg, rather than make the interfaces and register routes etc yourself.

You do still have to run kube-proxy otherwise you can't resolve service IPs though.

stv0g commented 3 years ago

You do still have to run kube-proxy otherwise you can't resolve service IPs though.

Maybe just setting a route for the service CIDR towards an in-cluster node/kube-proxy would be sufficient?

squat commented 3 years ago

Maybe just setting a route for the service CIDR towards an in-cluster node/kube-proxy would be sufficient?

That's very true, this will be sufficient. We can absolutely point the entire service CIDR at a single location leader. In fact, this is the same advice our docs currently give for routing service IPs from VPN peers. It will produce suboptimal routes whenever the pods in a service are scheduled in more than one location, however this might be a reasonable first iteration.

SerialVelocity commented 3 years ago

Yeah, that didn't work for me because when that node reboots/crashes/becomes unstable, you lose connectivity. kube-proxy is super easy to run though so it wasn't really an issue.