search

vmware-archive / kube-prod-runtime

A standard infrastructure environment for Kubernetes
Apache License 2.0
764 stars 134 forks source link

nginx-ingress service: externalTrafficPolicy: "Local" causes routing problems #1059

Open floek opened 3 years ago

floek commented 3 years ago

Hi,

I'm on platform generic with metallb and tried to activate the oidc authentication plugin with kube-prods key cloak. This didn't work because the kube-apiserver could not communicate with keycloaks nginx-ingress. I analysed this further and found the following issue: https://github.com/metallb/metallb/issues/153.

The issue comes up, when services published with externalTrafficPolicy: "Local" should be accessed from within the cluster. This doesn't work with kube-proxy in ipvs mode.

After some debugging I found, the nginx-ingress service is created from kube-prod with externalTrafficPolicy: "Local" enabled. As the kube-prod services published with the built in nginx-ingress are used for management, maybe it's better to set externalTrafficPolicy to Cluster. In my case I'm using a second nginx-ingress controller with an ingress class for the external services where I may need the client IPs.

Please consider to set externalTrafficPolicy to Cluster.

Cheers, floek

javsalgar commented 3 years ago

Hi,

Thank you so much for the input! I opened this PR https://github.com/bitnami/kube-prod-runtime/pull/1060 for discussion and to see how it works in our tests.