search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
48 stars 88 forks source link

Encrypting pod network traffic between worker node and pod VM #688

Open bpradipt opened 1 year ago

bpradipt commented 1 year ago

The default networking solution extends pod network by creating a VXLAN tunnel between worker node and pod VM. Does this need to be encrypted? If yes what are the available mechanisms ?

bpradipt commented 1 year ago

Few additional questions:

  1. When pod-to-pod traffic is secured via mTLS using service mesh, does it makes sense to encrypt the tunnel traffic ?
  2. If the workload is TLS enabled then does it makes sense to encrypt the tunnel traffic ?

As for the options to encrypt the tunnel traffic, I could find ipsec and macsec

The fundamental question is do we really need to encrypt the tunnel traffic ?

yoheiueda commented 1 year ago

I think it depends on how Kubernetes pod network is configured.

In a typical case, inter-node communication of Kubernetes pod network is not encrypted. In this case, we don't need to encrypt the tunnel traffic.

If inter-node communication of Kubernetes pod network is encrypted, the tunnel traffic should be encrypted. For example, Calico supports encryption of in-cluster pod traffic. https://docs.tigera.io/calico/3.25/network-policy/encrypt-cluster-pod-traffic

yoheiueda commented 1 year ago

WireGuard is gaining popularity for secure tunnels.

It looks like we can implement WireGuard tunnels for CAA in a relatively straightforward way.

https://www.wireguard.com/quickstart/ https://www.wireguard.com/netns/

bpradipt commented 1 year ago

I think it depends on how Kubernetes pod network is configured.

In a typical case, inter-node communication of Kubernetes pod network is not encrypted. In this case, we don't need to encrypt the tunnel traffic.

If inter-node communication of Kubernetes pod network is encrypted, the tunnel traffic should be encrypted. For example, Calico supports encryption of in-cluster pod traffic. https://docs.tigera.io/calico/3.25/network-policy/encrypt-cluster-pod-traffic

@yoheiueda this makes sense. We can provide it as an optional capability for cases where pod-to-pod traffic is not encrypted by default. For cases where pod-to-pod traffic is encrypted we can use wireguard.

I'm assuming mTLS using service mesh should work as-is. Although I have personally not tested service-mesh on a peer-pods environment.

huoqifeng commented 1 year ago

I ever tried istio service mesh for PeerPod, yes, it works as-is because mTLS configuration all happen within service mesh proxy.

bpradipt commented 1 year ago

I ever tried istio service mesh for PeerPod, yes, it works as-is because mTLS configuration all happen within service mesh proxy.

Thanks for confirming @huoqifeng

kartikjoshi21 commented 1 year ago

cc @mkulke

mkulke commented 1 year ago

Are we confident a tunnel over tunnel (wg over vxlan) configuration will cause no issues in some setups?

yoheiueda commented 1 year ago

@mkulke

WireGuard is a protocol that encapsulates IP packet as described here.

https://www.wireguard.com/#conceptual-overview

WireGuard securely encapsulates IP packets over UDP. You add a WireGuard interface, configure it with your private key and your peers' public keys, and then you send packets across it. All issues of key distribution and pushed configurations are out of scope of WireGuard; these are issues much better left for other layers, lest we end up with the bloat of IKE or OpenVPN

So, I don't think we need to encapsulate VXLAN over WireGuard for our pod network tunneling. We can just use WireGuard as a point-to-point network tunnel.

@kartikjoshi21 have you started implementing this enhancement?

BTW, I am working on refactoring our pod network code by simplifying the interface of the bitops package.

kartikjoshi21 commented 1 year ago

@yoheiueda I started trying out wireguard, But got stuck due to some issues in connectivity from worker node -> pod vm. Yet to look into what was missing in the setup.

yoheiueda commented 1 year ago

@kartikjoshi21 Thank you for the status info. I think I can help you if you can share the code or experimental setup information.

As I said before, I am refactoring the network code. (#947 and #954) This is probably conflicting with your code, and I can also help you to rebase your code if my PRs get merged before yours.

yoheiueda commented 1 month ago

I recently investigated the feasibility of WireGuard tunnel. I did some experiments using the ip command to set up WireGuard-based tunneling for peer pods, and it worked fine.

I also noticed that WireGuard has additional benefits for networks with NAT. If a worker node is a WireGuard client, and a peer pod VM is a WireGuard server, the worker node can be behind a network with NAT. https://www.wireguard.com/quickstart/#nat-and-firewall-traversal-persistence

This will work fine with the public IP support by this PR. https://github.com/confidential-containers/cloud-api-adaptor/pull/2056

The current VXLAN tunnel does not support NAT traversal, since Linux connection tracking cannot correcty handle VXLAN packet streams. https://github.com/confidential-containers/cloud-api-adaptor/issues/2015

yoheiueda commented 1 month ago

My current design of WireGuard support

WireGuard

katexochen commented 1 month ago

Why do we want to encrypt that traffic again? In the coco attacker model, the worker node is untrusted. So we are building a very secure channel to a party we don't trust anyway, protecting against what?

yoheiueda commented 1 month ago

@katexochen Yes. Worker nodes are not trusted, so we should not rely only on network encryption between a worker node and a peer pod VM. Applications runs on a pod need to use other protection mechanisms such as TLS for protecting data-plane traffic against worker nodes.

This optional feature is for an additional protection as defense in depth. The data-plane tunnel encryption can protect attacks from outside a Kubernetes cluster. In some use cases such as https://github.com/confidential-containers/cloud-api-adaptor/pull/2056 , tunnel traffic will go through the Internet. This additional protection may mitigate risks of data leakage when basic data protection is not properly configured.

Another benetit of WireGuard is the support of NAT traversal as mentioned in https://github.com/confidential-containers/cloud-api-adaptor/issues/688#issuecomment-2407028089. VXLAN does not work well with NAT, and WireGuard will also be an option to support such network topologies.

@bpradipt @davidhadas Any comments?

bpradipt commented 1 month ago

@katexochen this is a vxlan tunnel alternative. One of the use cases that we have for cloud-api-adaptor is bursting to cloud from a development environment (eg kind cluster on a dev laptop) to run large VM or GPU VM to experiment with AI models. Another use case that we have is enterprise focused where it is used for bursting from on-prem to cloud. Although in this case there is site-to-site vpn but we have found issues with VXLAN tunneling.

katexochen commented 1 month ago

So we expected better tunnel stability from switching to wg and won't nest it?

yoheiueda commented 1 month ago

So we expected better tunnel stability from switching to wg

Yes.

and won't nest it?

Does "Nest" mean "WireGuard over VXLAN" or "VXLAN over WireGuard"? No. We don't need to nest the two protocols to support encryption. WireGuard is a single protocol that support both encryption and tunneling. It is different from nested protocols like "VXLAN over IPSec".

yoheiueda commented 1 week ago

I prepared slides of my new proposal for this issue. Peer Pod Network Tunneling.pdf