search

cilium / cilium

eBPF-based Networking, Security, and Observability
https://cilium.io
Apache License 2.0
19.16k stars 2.78k forks source link

make Cilium to optionally prefer external IPs #31902

Open errordeveloper opened 2 months ago

errordeveloper commented 2 months ago

I am looking to configure Cilium on a cluster where some nodes cannot communicate over internal IPs, and I don't wish to use a VPN or some kind of VPC peering, I'd rather just use Cilium with WireGuard to provide this connectivity.

I can see that one option is to pass --ipv4-node, but I would need to do it on each node individually and cannot use the helm chart for that. Also, this option only takes a single value, and I'd like to configure WireGurad in a way that it picks public or private IPs as needed, however, it doesn't seem currently possible. I can see that CiliumNode already holds multiple addresses per node, however as far as tunnel setup is concerned, it comes down to a single address.

So just to clarify, it'd be very nice if I could have WireGuard (or VXLAN) tunnels spanning public and private networks. E.g. some nodes can communicate privately, while others need to be accessed over a public network.

errordeveloper commented 2 months ago

What could help as a stop-gap would be --ipv4-node=external, and perhaps some way of setting this as node annotation or as a node label selector.

errordeveloper commented 2 months ago

In terms of configuration, I think the ideal scenario would be to have node labels that specify partition ID. Cilium would pick the right addresses to use across partitions with different IDs.

errordeveloper commented 2 months ago

Another easy solution might be to make Cilium include external addresses in WG config. I tried to patch it manually on the nodes with wg setconf, but looks like Cilium reconciles the config, as understandably it should.

joestringer commented 2 months ago

One thing I will point out with respect to VXLAN tunnels spanning public networks is that this could significantly increase the security risks for man-in-the-middle and network attackers as per the Threat Model. As such, I would want to ensure that (a) Cilium continues to have secure-by-default settings and (b) we provide sufficient warning and clear configuration for users who understand the implications and wish to proceed.

jspaleta commented 2 months ago

hey, so I'm definitely interested in this. However like joe, I have a security concern. Speaking of the WireGuard tunnel specifically, how do you bootstrap the sharing of the wg peer public keys securely between nodes in your hybrid cluster?

Are you mandating as an implicit requirement in the supported configuration that control plane nodes must have kapi service accessible to all nodes prior to the establihment of the wg tunnel? In a hybrid cluster, I guess that would mean the control plane nodes must expose a global routable kapi address and can't be on a private network.

For comparison, I'm looking at how Talos Linux accomplishes this and they end up relying on a separate discovery service mechanism and specialized client code in their distribution to make requests to this discovery service as part of node provisioning so that control plane nodes don't have to be on the public network... just the wireguard peer discovery service.

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.