search

nokia / danm

TelCo grade network management in a Kubernetes cluster
BSD 3-Clause "New" or "Revised" License
373 stars 81 forks source link

danm/calico: pod to pod communication does not traverse nodes #265

Closed ololdach closed 2 years ago

ololdach commented 2 years ago

possible bug

What happened: We set up two k3s 1.21.7 clusters with multiple master/worker nodes each. One with danm 4.3.0 and calico as bootstrap network, one with pure calico. When we started to deploy workloads we noticed that all works fine, as long as no NetworkPolicies are in place and the default allow-all policy is being applied. As soon as a label based NetworkPolicy is introduced in a namespace, the pakets in the default cn will no longer traverse nodes. Only pods on the same node can communicate. Interestingly port based policies seem to work. If a policy allows access based on ports/ips and without label matching, the traffic is fine and works as expected. We traced it down to the iptables chain that would allow the pakets to pass having no (0) references. Our guess is, that some metadata that would be needed in the lower level CNIs to hook into label matching rules is not being passed by danm when it comes to NetworkPolicies. In our pure calico reference cluster, everything works as expected.

What you expected to happen: We expected that pod-pod traffic works transparently across nodes with NetworkPolicies when using calico as bootstrap network

How to reproduce it: Install danm on top of Calico in a cluster with two nodes. Create two namespaces, no-policy and policy. In both namespaces, deploy two debug/ubuntu pods with a name label assigned. Test the communication in the no-policy namespace by pinging the pods and expect it to work. Assign a NetworkPolicy to the policy namespace based on the name label of the pod, allowing them to communicate. Try to ping across the nodes... and wait for the timeout.

Environment:

ololdach commented 2 years ago

Filling in the form really made me forget to mention the things of utmost importance: danm is great stuff and we're grateful for the awesome project. It really helps us solve some tricky issues and we're happy to help making it even better. Thanks for your help!

ololdach commented 2 years ago

Update: Only Label matching policies are affected. Port based work fine.

Levovar commented 2 years ago

As soon as a label based NetworkPolicy is introduced in a namespace, the pakets in the default cn will no longer traverse node

as in, already deployed Pods suddenly cut off, or Pods deployed after the NP have their connectivity set up wrong? asking cause at first glance this seems kinda interesting. as far as I'm aware Calico -at least used to- provision iptables rules via a separate DaemonSet, into the host's networking namespace. i.e. CNI is not involved in this chain if your existing Pods are cut off that would indicate something is up in the Calico control plane, cause no CNI action is invoked in this case (there is only CNI ADD and CNI DEL)

sure it still might be something got lost during the initial CNI operation and later on it caused an unintended interaction, but I think it might be a more complicated case. can you share the answer above, your Calico CNI config file, NetworkPolicy API used (K8s? Calico?), iptables delta after adding the NP

ololdach commented 2 years ago

It happens in newly deployed pods at creation time. Usually they fail to come up when they rely on communication to other entities or they end up defunct. I'm not sure I understand what you mean by NP have their connectivity set up wrong. From my side it looks as if the NPs are applied to the pods at creation time, resulting in the required iptable chains to be created by calico. Also, only label based rules are affected. Could it be, that the label metadata is not being passed down to calico? NP API: k8s

calico.conf: { "cniVersion": "0.3.1", "name": "calico-pod-network", "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "katana", "mtu": 0, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "container_settings": { "allow_ip_forwarding": true }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }

As for the delta, please give me a moment to collect it.

ololdach commented 2 years ago

Levovar, I apologise for raising this issue prematurely. Just in case that somebody else runs into this setup, I'll detail a little about our findings, before I ask you to close this as resolved aka not related to danm.

The observed problem is: Two pods that have a label based network policy allowing them to communicate will not be allowed to talk to each other over the pod network, once they are on two different nodes. The root cause is that the iptable rules that would allow the pakets to pass are either missing or disconnected. However, there are multiple reasons:

The good news is: It's definitely not related to danm!

Cheers Oliver

Levovar commented 2 years ago

thanks for the detailed case study Oliver, appreciate it! gonna close the issue then :)