Service translation problem when source pod contact a loadbalancerIP on the same cluster

faelis commented 3 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

I have a use case where a pod need to access another pod in a different namespace through the loadbalancerIP of its service. It works well until I add a cnp that disallow the communication between nodes. With hubble, I can see that the flow is between the source pod and the destination, and the cnp deny it. It looks like cilium translate the loadbalancerIP like a clusterIP. If I'm right, how to disable this translation when the destination IP is a loadbalancerIP on the same cluster without allowing specific flow with a cnp?

Cilium Version

tested with 1.15.6 and 1.16.0-rc.1

Kernel Version

5.15.158.2-1.cm2

Kubernetes Version

v1.28.3

Regression

No response

Sysdump

No response

Relevant log output

No response

Anything else?

With the latest version, I set the disable-external-ip-mitigation to true, with no luck.

Cilium Users Document

[X] Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

[X] I agree to follow this project's Code of Conduct

joestringer commented 3 months ago

Can you also provide an example policy that you applied, and hubble observe output that demonstrates the policy deny? This could help to point out exactly how the policy engine is handling this traffic.

I have a suspicion that the LB traffic is translated into a NodePort and subsequently at the destination node, translated into the backend traffic, as part of standard k8s service design.. in which case the traffic would truly be reaching another Node in the cluster. Hubble output could help to confirm this. If there are IP addresses, please also highlight what the IP addresses are (eg node IP or Pod IP).

faelis commented 2 months ago

Hi @joestringer, sorry for the delay.

No need for networkpolicy to demonstrate the 'unwanted' translation.

Here is our implementation :

We are on AKS.
The loadbalancer IPs are taken from the same subnet as the one used by the nodes.
kubectl cnp,ccnp,netpol -A
kubectl get nodes
kubectl get services -n dok-dev01
kubectl get pods -n dok-dev01

I run the test like this :

kubectl run -ti test --image curlimages/curl:8.7 -n dok-dev02 -- sh
while true; do curl -k https://10.131.180.27; sleep 3; done

I run hubble like this : hubble observe flow --identity 28586 -- identity 9784 (28586 is the identity of the test pod and 9784 is the identity of the nginx ingress controller pods)

The curl is against the loadbalancerIP/ExternalIP of the kubernetes service in front of the nginx ingress controller from a pod named test. But cilium see the flow between the test pod and the nginx ingress controller pod. I would like to disable this 'translation', and see flow like this : test pod -> loadbalancerIP (world) -> nginx ingress controller pod

joestringer commented 2 months ago

Could you speak a little bit more about the use case, why is it important to you that these flows are not translated at the source?

faelis commented 2 months ago

We manage large multitenant cluster :

Each tenant has its own namespace
Some of our tenant want to expose services (for exemple, one could expose a keycloak or a postgres)
The tenants don't (and can't) know their neighbors
We don't want to allow each flow between each tenant when they need it
We need to control the ingress and the egress inside the cluster
We need some network policy that isolate each tenant namespace

If the consumer is outside the cluster, the flow is as wanted. If the consumer is inside the cluster, the translation is made, and the namespace isolation disallow the flow.

That the reasons why we don't want the flow to be translated if the destination ip is an externalIP/loadbalancerIP of the same cluster. And I thought that the disable-external-ip-mitigation feature will help us, maybe I was wrong.

cilium / cilium