GKE Load Balancer - Proxy protocol with whitelist-source-range

[Loksonick]

What happened: When accessing vbe-ingress endpoint, nginx logs are:

Running curl twice I get diffferent client IPs:

curl 35.195.68.XX/voice-bot-entrypoint/docs

2024/02/28 18:44:40 [error] 179#179: *80656 broken header: "GET /voice-bot-entrypoint/docs HTTP/1.1
Host: 35.195.68.XX
User-Agent: curl/7.81.0
Accept: */*

" while reading PROXY protocol, client: 85.222.102.XXX, server: 0.0.0.0:80
2024/02/28 18:45:19 [error] 178#178: *80969 broken header: "GET /voice-bot-entrypoint/docs HTTP/1.1
Host: 35.195.68.19
User-Agent: curl/7.81.0
Accept: */*

" while reading PROXY protocol, client: 10.132.0.6, server: 0.0.0.0:80

10.132.0.6 client IP is assigned to those pods:

$ kubectl get all -A -o wide | grep 10.132.0.6
kube-system       pod/anetd-lvgns                                               1/1     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/filestore-node-jzxxg                                      3/3     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/fluentbit-gke-big-vx4g6                                   2/2     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/gke-metadata-server-bbjvr                                 1/1     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/gke-metrics-agent-wfxld                                   2/2     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/ip-masq-nrf-876d6                                         1/1     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/netd-mvl5s                                                1/1     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/node-local-dns-snzwt                                      1/1     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>
kube-system       pod/pdcsi-node-g9m89                                          2/2     Running     0               34h     10.132.0.6      gk3-nzoz-k-xxxx-nap-mt866437-19284e13-k977   <none>           <none>

$ kubectl describe cm -n ingress-nginx ingress-nginx-controller
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.9.6
              helm.sh/chart=ingress-nginx-4.9.1
              k8slens-edit-resource-version=v1
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
allow-snippet-annotations:
----
true
use-proxy-protocol:
----
true

What you expected to happen: Real client IP is available, whitelisting works.

NGINX Ingress controller version :

NGINX Ingress controller
  Release:       v1.9.6
  Build:         6a73aa3b05040a97ef8213675a16142a9c95952a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.8-gke.1067004

Environment:

• Cloud provider or hardware configuration: GKE

• OS (e.g. from /etc/os-release): Linux

• Kernel (e.g. uname -a): Linux microsoft-standard-WSL2

• Install tools: GKE private cluste with Autopilot

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

    helm ls -A | grep -i ingress
    ingress-nginx     ingress-nginx   1   2024-02-28 11:34:55.637619574 +0100 CET deployed    ingress-nginx-4.9.1   1.9.6

  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

    USER-SUPPLIED VALUES:
    controller:
    allowSnippetAnnotations: true
    config:
    use-proxy-protocol: "true"
    service:
    internal:
    annotations:
      networking.gke.io/load-balancer-type: Internal
    enabled: true

• Current State of the controller:

  • kubectl describe ingressclasses

    Name:         nginx
    Labels:       app.kubernetes.io/component=controller
          app.kubernetes.io/instance=ingress-nginx
          app.kubernetes.io/managed-by=Helm
          app.kubernetes.io/name=ingress-nginx
          app.kubernetes.io/part-of=ingress-nginx
          app.kubernetes.io/version=1.9.6
          helm.sh/chart=ingress-nginx-4.9.1
    Annotations:  meta.helm.sh/release-name: ingress-nginx
          meta.helm.sh/release-namespace: ingress-nginx
    Controller:   k8s.io/ingress-nginx
    Events:       <none>

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide

    
    kubectl -n ingress-nginx get all -o wide
    NAME                                            READY   STATUS    RESTARTS   AGE   IP            NODE                                    NOMINATED NODE   READINESS GATES
    pod/ingress-nginx-controller-5f4f6d5bfb-j2gsf   1/1     Running   0          18m   10.49.0.156   gk3-nzoz-k-xxxx-pool-2-5436578e-85lk   <none>           <none>

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller LoadBalancer 34.118.225.XX 35.195.148.XXX 80:30827/TCP,443:31256/TCP 22m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller-admission ClusterIP 34.118.234.XXX 443/TCP 22m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller-internal LoadBalancer 34.118.229.XXX 10.132.0.18 80:32510/TCP,443:30674/TCP 22m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 1/1 1 1 22m controller registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-5f4f6d5bfb 1 1 1 22m controller registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5f4f6d5bfb

  - `kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>`

kubectl -n ingress-nginx describe po ingress-nginx-controller-5f4f6d5bfb-j2gsf Name: ingress-nginx-controller-5f4f6d5bfb-j2gsf Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: gk3-nzoz-k-xxxx-pool-2-5436578e-85lk/10.132.0.5 Start Time: Wed, 28 Feb 2024 11:38:46 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.6 helm.sh/chart=ingress-nginx-4.9.1 pod-template-hash=5f4f6d5bfb Annotations: Status: Running SeccompProfile: RuntimeDefault IP: 10.49.0.156 IPs: IP: 10.49.0.156 Controlled By: ReplicaSet/ingress-nginx-controller-5f4f6d5bfb Containers: controller: Container ID: containerd://471a28ef7dc45229ba896f5d62349f2bce29ce77e98691a67e32db9c9cfda82a Image: registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c Image ID: registry.k8s.io/ingress-nginx/controller@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Wed, 28 Feb 2024 11:38:47 +0100 Ready: True Restart Count: 0 Limits: cpu: 250m ephemeral-storage: 1Gi memory: 512Mi Requests: cpu: 250m ephemeral-storage: 1Gi memory: 512Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5f4f6d5bfb-j2gsf (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4kt6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-f4kt6: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: kubernetes.io/arch=amd64:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Normal Scheduled 24m gke.io/optimize-utilization-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5f4f6d5bfb-j2gsf to gk3-nzoz-k-xxxx-pool-2-5436578e-85lk Normal Pulled 24m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c" already present on machine Normal Created 24m kubelet Created container controller Normal Started 24m kubelet Started container controller Normal RELOAD 24m nginx-ingress-controller NGINX reload triggered due to a change in configuration

  - `kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>`

$ kubectl -n ingress-nginx describe svc ingress-nginx-controller Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.6 helm.sh/chart=ingress-nginx-4.9.1 Annotations: cloud.google.com/neg: {"ingress":true} meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 34.118.225.XX IPs: 34.118.225.XX LoadBalancer Ingress: 35.195.148.XXX Port: http 80/TCP TargetPort: http/TCP NodePort: http 30827/TCP Endpoints: 10.49.0.156:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 31256/TCP Endpoints: 10.49.0.156:443 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message

---

Normal ADD 30m sc-gateway-controller ingress-nginx/ingress-nginx-controller Normal EnsuringLoadBalancer 30m service-controller Ensuring load balancer Normal UPDATE 30m sc-gateway-controller ingress-nginx/ingress-nginx-controller Normal DNSRecordProvisioningSucceeded 30m (x4 over 30m) clouddns-controller DNS records updated Normal EnsuredLoadBalancer 29m service-controller Ensured load balancer

$ kubectl -n ingress-nginx describe svc ingress-nginx-controller-admission Name: ingress-nginx-controller-admission Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.6 helm.sh/chart=ingress-nginx-4.9.1 Annotations: cloud.google.com/neg: {"ingress":true} meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 34.118.234.XXX IPs: 34.118.234.XXX Port: https-webhook 443/TCP TargetPort: webhook/TCP Endpoints: 10.49.0.156:8443 Session Affinity: None Events: Type Reason Age From Message

---

Normal ADD 30m sc-gateway-controller ingress-nginx/ingress-nginx-controller-admission Normal DNSRecordProvisioningSucceeded 30m (x4 over 30m) clouddns-controller DNS records updated

$ kubectl -n ingress-nginx describe svc ingress-nginx-controller-internal Name: ingress-nginx-controller-internal Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.6 helm.sh/chart=ingress-nginx-4.9.1 Annotations: cloud.google.com/neg: {"ingress":true} meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx networking.gke.io/load-balancer-type: Internal Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 34.118.229.XXX IPs: 34.118.229.XXX LoadBalancer Ingress: 10.132.0.18 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32510/TCP Endpoints: 10.49.0.156:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30674/TCP Endpoints: 10.49.0.156:443 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message

---

Normal ADD 31m sc-gateway-controller ingress-nginx/ingress-nginx-controller-internal Normal EnsuringLoadBalancer 31m service-controller Ensuring load balancer Normal UPDATE 31m (x2 over 31m) sc-gateway-controller ingress-nginx/ingress-nginx-controller-internal Normal DNSRecordProvisioningSucceeded 31m (x4 over 31m) clouddns-controller DNS records updated Normal EnsuredLoadBalancer 29m service-controller Ensured load balancer

- **Others**:
    - `kubectl describe cm -n ingress-nginx ingress-nginx-controller`

Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.6 helm.sh/chart=ingress-nginx-4.9.1 k8slens-edit-resource-version=v1 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx

Data

allow-snippet-annotations:

true use-proxy-protocol:

true

**How to reproduce this issue**:
1. Create new GKE private cluster with Autopilot. 
2. Install ingress-nginx `helm upgrade --install ingress-nginx ingress-nginx   --repo https://kubernetes.github.io/ingress-nginx   --namespace ingress-nginx -f values.yaml`
3. Create any deployment which can handle HTTP requests. 
4. Create Ingress like:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: your-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/whitelist-source-range: "xxx.xxx.xxx.xxx" # I provide correct IP's here spec: ingressClassName: nginx rules:

• http: paths:
  • path: {{ .Values.root_path }}(/|$)(.*) pathType: Prefix backend: service: name: your-service port: number: 80

    
    5. Try to connect to service using whitelisted IP.

Anything else we need to know: Just so you know, I'm not a networking/devops expert. I am probably making a simple mistake that I will be ashamed of later.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

/remove-kind bug

• Please show proof screenshot that proxy-protocol is enabled on GCP LB
• Please show proof kubectl command output that proxy-protocol is enabled on controller
• Please remove the yaml info and please show the kubectl describe output for the related resources
• You have not answered many questions asked in the template so please answer them in MD format
• Please show real actual curl command used to make HTTP request
• Please show the actual logs of the controller pod so the logmessages of the curl related request are visible

[longwuyuan]

/triage needs-information

[Loksonick]

• Please show proof screenshot that proxy-protocol is enabled on GCP LB I can't edit GCP load balancer, poxy-protocol checkbox is not visible (I was following nginx docs, end up here, but proxy-protocol wasn't visible)

• Please show proof kubectl command output that proxy-protocol is enabled on controller Issue edited. kubectl describe cm -n ingress-nginx ingress-nginx-controller is fine?

• Please remove the yaml info and please show the kubectl describe output for the related resources There is ingress-nginx pod, configmap and services describe, anything else?

• You have not answered many questions asked in the template so please answer them in MD format I think those unanswered questions werre answered in other bulletpoint, anything more than present is essential?

• Please show real actual curl command used to make HTTP request Done.

• Please show the actual logs of the controller pod so the logmessages of the curl related request are visible I think that was done before but updated it anyway.

[longwuyuan]

• Your clusterIP addresses are valid C class internet ipaddresses. I don't understand
• Without knowing that GCP LB has been enabled for proxy-protocol, I am not sure what to do

[Loksonick]

It seems I don't understand something. I don't use loadbalancer creted "by" GCP. If uninstall nginx-controller from my cluster there will be no loadbalancers listed in GCP thus I assume the only LB I have is the nginx one created in cluster.

[longwuyuan]

• GKE private cluster with Autopilot is out of scope of discussion here
• Ingress Controllers create Kubernetes Objects
• One of the K8S objects created by the ingress-nginx controller is a K8S object of "kind: service" of "--type LoadBalancer"
• Anytime anyone creates a service of --type LoadBalancer in Kubernetes, it provisions a LoadBalancer object on the infra-provider like AWS/GCP/Azure/Other. If on-premises, then most often there is a software like Metallb.org which provides the same functionality as a LB from AWS/GCP/Azure
• Since you made the comment above and after considering your values file spec "networking.gke.io/load-balancer-type: Internal", I think you should delete the installation of the ingress-controller and make sure all resources related to the ingress-nginx controller are deleted
• Then I think you should follow the documentaiton https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke
• I also think you should install one controller for external and another conroller for internal as per this https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster
• And I think you should enable proxy-protocol on the GCP LB as per https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#proxy-protocol
• And you already know how to enable proxy-protocol on the controller https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol

[longwuyuan]

/assign

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.