search

kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Support nodes without public IP address on Packet to reduce cluster exposure #277

Open rata opened 4 years ago

rata commented 4 years ago

Packet servers now provide the functionality to specify the address type that you want (privateIPv4, public_IPv4, public_IPv6) in any combination you want.

This can be set using terraform in the packet_device resource: https://www.terraform.io/docs/providers/packet/r/device.html (look for ip_address_types).

The goal of this issue is to create Lokomotive clusters with workers pools that have only private IPv4 addresses.

Don't know if this will "just work", maybe node will not have internet access, for example, if it configured this way and it might be problematic. We need to check when tackling this issue

surajssd commented 4 years ago

Q: Calico still creates the BGP setup over public interface right? If we disable the public IP will it conflict with metallb BGP?

We don't have any e2e tests to verify such things. Above scenario will have to tested manually.

rata commented 4 years ago

Q: Calico still creates the BGP setup over public interface right? If we disable the public IP will it conflict with metallb BGP?

No, calico is using the private interface IIRC. And, in that case, no changes expected and all should work.

I think no clashes as MetalLB will be active, initiating the connection, to the right peers in Packet. It's not that packet peers will initiate the connection to the nodes (therefore reaching calico). So, I don't expect any issues.... but maybe I'm missing something :)

johananl commented 4 years ago

IIUC a node without a public IP will be completely isolated from the internet. Packet doesn't provide a "NAT gateway" functionality, so although the node could reach the Packet ToR switch, it won't be able to talk to internet hosts using its private IP. This is a problem because, for example, that node won't be able to download container images or Flatcar updates.

A possible solution to this could be to use a designated "gateway node" in the cluster, i.e. one node which has a public IP and allows nodes to communicate with the internet.

rata commented 4 years ago

I tried again manually, using Packet UI, and it showed this error now (yesterday just ignored and created with IPv6 and public IPv4): Public IPv4 is needed for flatcar_stable

So, I don't think we can do anything but remove IPv6 without getting in touch with Packet