search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.01k stars 1.34k forks source link

When eBPF is enabled, when ippool natOutgoing set to false connection to kubernetes api through serviceIP starts timing out #8829

Open tomastigera opened 5 months ago

tomastigera commented 5 months ago

Noticed one different behavior when eBPF is enabled, when ippool natOutgoing set to false connection to kubernetes api through serviceIP starts timing out from pods wanting to use it, since there is no iptables rule sending service ip out to correct node.

How does this work in eBPF mode ?

I am forced to enable outgoing nat to solve this issue, is this a bug or expected behavior?

Originally posted by @ehsan310 in https://github.com/projectcalico/calico/issues/8812#issuecomment-2113624733

tomastigera commented 5 months ago

@ehsan310 Are your API servers on a different network,not reachable without MASQ or overlay?

since there is no iptables rule sending service ip out to correct node.

Destination IP is changed to whatever the service is translated to and then the packet is sent out. It must be routeable. It could be that kube-proxy does the MASQ on its own based on your cluster configuration, i.e. it knows that the traffic is leaving the pod network, and calico ebpf does not do it because it does not have that configuration - nat outgoing serves that purpose.

ehsan310 commented 5 months ago

service ip is advertised via bgp and calico, there is no overlay network and I have a flat networking peered with ToR. also kube-proxy is disable and DS is removed , all old iptable rules are also removed. so it's only calico handling the traffic. Node to Node Mesh is also disabled.

that also make sense based on what you said , because we have separate mgmt and pod/service traffic network and k8s api is listening on mgmt network , so this is expected.

ehsan310 commented 5 months ago

I also hit this issue https://github.com/projectcalico/calico/issues/5039

tomastigera commented 1 month ago

Sorry for putting this issue on aback burner, did you make any progress with it?

ehsan310 commented 1 month ago

Sorry for putting this issue on aback burner, did you make any progress with it?

I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible and with a installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api that translate to host network until BGP is established which is not available in during the initialization.

tomastigera commented 1 week ago

I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible

Do you mean that kubernetes-services-endpoint config map? You can do that for manifest as well.

installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api

Do you mean SNAT? How did that work with iptables?

ehsan310 commented 1 week ago

I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible

Do you mean that kubernetes-services-endpoint config map? You can do that for manifest as well.

installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api

Do you mean SNAT? How did that work with iptables?

oh really ? I thought that CM has only been used in tigera-operator pod ! the issue was when deploying calico via kubespray (which uses manifest) and disable kube-proxy then pods are unable to communicate to kube-api endpoint since there is no kube-proxy in place (no snat, or any iptables rule) then until calico-node pod starts nothing will work and the issue was calico-api-server and calico-kube-controllers were trying to connect to kube-api but was not able to reach it. in operator kubernetes-services-endpoint i pointed to master nic ip and port 6443 directly to make the connection. does that make sence ?

tomastigera commented 1 week ago

oh really ? I thought that CM has only been used in tigera-operator pod !

you can also put it straight into the manifest as env vars. That was the original way of doing it. Otherwise, as you are describing, our ebpf kube-proxy replacement would not be able to reach api server and implement the services. Bit of an chicken-egg problem. We are going to make it more automated with operator.