search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.47k stars 8.25k forks source link

AWS NLB health proxy protocol broken header #6996

Closed panho66 closed 2 years ago

panho66 commented 3 years ago

I have AWS NLB with one target group TLS port 443 forward to EC2 running Nginx reverse proxy. The NLB target group has Proxy protocol v2 enabled. I also enable proxy protocol support in Nginx and everything work. I could browser application via NLB.

However, I found that Nginx has error message in log fie

2021/03/25 00:50:06 [error] 36#36: 225344 broken header: "" while reading PROXY protocol, client: 10.42.1.0, server: 0.0.0.0:443 2021/03/25 00:50:07 [error] 36#36: 225349 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443 2021/03/25 00:50:08 [error] 36#36: 225355 broken header: "" while reading PROXY protocol, client: 10.42.1.0, server: 0.0.0.0:443 10.139.219.52 - - [25/Mar/2021:00:50:09 +0000] "GET /app/shoppingcart/add.do HTTP/1.1" 500 3056 "-" "curl/7.29.0" 115 0.008 [shoppingcart-billing-http] [] 10.42.1.80:8080 3056 0.008 500 d3ceskjfjsdjhfohewerwer963790 2021/03/25 00:50:09 [error] 36#36: 225362 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443 2021/03/25 00:50:09 [error] 36#36: *225363 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443

I am using kubernetes-ingress-controller/nginx-ingress-controller:0.32.0 I believe that error message causes by NLB TCP health check.

I install nginx via helm value.yaml

podSecurityPolicy: enabled: "true" controller: config: real-ip-header: "proxy_protocol" use-proxy-protocol: "true" server-tokens: "False" proxySetHeaders: X-Request-Start: "t=${msec}" service: annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-southeast-2:123343434:certificate/dsfsddf-sdfsd-sdfsdf-sdfsfd-sdfsdfdsfsdfsf service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-FS-1-2-Res-2019-08 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-type: nlb service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" enableHttp: false nodePorts: http: 30000 https: 30001 loadBalancerSourceRanges:

k8s-ci-robot commented 3 years ago

@panho66: The label(s) triage/support cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/6996): >I have AWS NLB with one target group TLS port 443 forward to EC2 running Nginx reverse proxy. The NLB target group has Proxy protocol v2 enabled. I also enable proxy protocol support in Nginx and everything work. I could browser application via NLB. > >However, I found that Nginx has error message in log fie > >2021/03/25 00:50:06 [error] 36#36: *225344 broken header: "" while reading PROXY protocol, client: 10.42.1.0, server: 0.0.0.0:443 >2021/03/25 00:50:07 [error] 36#36: *225349 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443 >2021/03/25 00:50:08 [error] 36#36: *225355 broken header: "" while reading PROXY protocol, client: 10.42.1.0, server: 0.0.0.0:443 >10.139.219.52 - - [25/Mar/2021:00:50:09 +0000] "GET /app/shoppingcart/add.do HTTP/1.1" 500 3056 "-" "curl/7.29.0" 115 0.008 [shoppingcart-billing-http] [] 10.42.1.80:8080 3056 0.008 500 d3ceskjfjsdjhfohewerwer963790 >2021/03/25 00:50:09 [error] 36#36: *225362 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443 >2021/03/25 00:50:09 [error] 36#36: *225363 broken header: "" while reading PROXY protocol, client: 10.42.2.0, server: 0.0.0.0:443 > >I am using kubernetes-ingress-controller/nginx-ingress-controller:0.32.0 >I believe that error message causes by NLB TCP health check. > >I install nginx via helm value.yaml > >--- > podSecurityPolicy: > enabled: "true" > controller: > config: > real-ip-header: "proxy_protocol" > use-proxy-protocol: "true" > server-tokens: "False" > proxySetHeaders: > X-Request-Start: "t=${msec}" > service: > annotations: > service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl > service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" > service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 > service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-southeast-2:123343434:certificate/dsfsddf-sdfsd-sdfsdf-sdfsfd-sdfsdfdsfsdfsf > service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-FS-1-2-Res-2019-08 > service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" > service.beta.kubernetes.io/aws-load-balancer-type: nlb > service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" > enableHttp: false > nodePorts: > http: 30000 > https: 30001 > loadBalancerSourceRanges: > - 0.0.0.0/8 > >/triage support > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
davidazael commented 3 years ago

I believe this might help you:

https://stackoverflow.com/questions/33620183/nginx-broken-header-with-proxy-protocol-and-elb

panho66 commented 3 years ago

@davidazael That doc document didn't help. I believe I have all configuration correctly. The problem seems AWS NLB health do not send proxy header and Nginx dump error

[error] 38#38: 1232 broken header: "" while reading PROXY protocol, client: 10.97.14.41, server: 0.0.0.0:443 [error] 38#38: 1317 broken header: "" while reading PROXY protocol, client: 10.42.1.0, server: 0.0.0.0:443

is there a way to suppress this error message only ? may be consider this is warn rather than error. What if health check is TCP open port without sending Proxy header string ?

panho66 commented 3 years ago

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol

Health check connections After you enable proxy protocol, the proxy protocol header is also included in health check connections from the load balancer. However, with health check connections, the client connection information is not sent in the proxy protocol header.

Is this nginx complain about ?

longwuyuan commented 3 years ago

Can you show helm ls -A

Thanks, ; Long

On Mon, 12 Apr, 2021, 10:55 AM panho66, @.***> wrote:

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol

Health check connections After you enable proxy protocol, the proxy protocol header is also included in health check connections from the load balancer. However, with health check connections, the client connection information is not sent in the proxy protocol header.

Is this nginx complain about ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/6996#issuecomment-817490970, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWVOMBUGZACR35BZ54TTIJ75TANCNFSM4Z4D5AAQ .

panho66 commented 3 years ago

@longwuyuan ah I am not using helm but Rancher catalog

kubectl get pod -A

nginx-ingress nginx-ingress-controller-6d6596d45b-49mzk 1/1 Running 4 3d21h nginx-ingress nginx-ingress-controller-6d6596d45b-5slsq 1/1 Running 3 3d21h nginx-ingress nginx-ingress-controller-6d6596d45b-t4q9w 1/1 Running 4 3d21h nginx-ingress nginx-ingress-default-backend-679d47bf54-mh9bm 1/1 Running 25 67d

chart: https://charts.helm.sh/stable chart 1.39.0 values.yaml

---
  podSecurityPolicy:
     enabled: "true"
  defaultBackend:
    image:
       repository: nexus3/air-gap/defaultbackend-amd64
  controller:
    image:
       repository: nexus3/air-gap/kubernetes-ingress-controller/nginx-ingress-controller
    admissionWebhooks:
       patch:
          image:
             repository: nexus3/air-gap/jettech/kube-webhook-certgen
    replicaCount:  "3"
    autoscaling:
      enabled: false
      maxReplicas: "3"
      minReplicas: "3"
    config:
      real-ip-header: "proxy_protocol"
      use-proxy-protocol: "true"
      server-tokens: "False"
    proxySetHeaders:
      X-Request-Start: "t=${msec}"
    resources:
      limits:
        cpu: "700m"
        memory: "768Mi"
      requests:
        cpu: "300m"
        memory: "500Mi"
    service:
      annotations:
         service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: owner=middleware,env:name=dev
         service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
         service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
         service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
         service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-southeast-2:1234567890:certificate/8fb14a7c-a721-4f87-ad15-85cb86c603e6
         service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-FS-1-2-Res-2019-08
         service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
         service.beta.kubernetes.io/aws-load-balancer-type: nlb
         service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
         service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
         service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
         service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "cf-logs-1234567890"
         service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "access-logs"
         nginx.ingress.kubernetes.io/force-ssl-redirect: true
      enableHttp: false
      nodePorts:
         http:  30000
         https: 30001
      loadBalancerSourceRanges:
               - 10.0.0.0/8
longwuyuan commented 3 years ago

kubectl describe pod nginx-ingress nginx-ingress-controller-6d6596d45b-49mzk

longwuyuan commented 3 years ago

Just FYI ;

panho66 commented 3 years ago

@longwuyuan sorry for the confusion. I am using Rancher catalog to deploy helm chart that why I do not need helm client install. Rancher has build in helm no need to install helm client. So other than deployment method, there is nothing related to Rancher function.

If your final goal with proxy-protocol is to get client-ip, then setting externalTrafficPolicy: Local is the trending spec of choice

Regarding to externalTrafficPolicy: Local, "Local" preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially imbalanced traffic spreading. "Cluster" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. So, I use externalTrafficPolicy: cluster

In order to get client IP address, I set up nginx to use proxy protocol enable-real-ip: true use-proxy-protocol: true

and also enable proxy protocal v2 in AWS NLB.

I just want to have optimum configuration... but seems Nginx dumping AWS health check error

> kubectl describe pod  nginx-ingress-controller-6d6596d45b-49mzk -n nginx-ingress
Name:         nginx-ingress-controller-6d6596d45b-49mzk
Namespace:    nginx-ingress
Priority:     0
Node:         ip-10-97-29-146.ap-southeast-2.compute.internal/10.97.29.146
Start Time:   Thu, 08 Apr 2021 07:57:30 +0000
Labels:       app=nginx-ingress
              app.kubernetes.io/component=controller
              component=controller
              pod-template-hash=6d6596d45b
              release=nginx-ingress
Annotations:  cattle.io/timestamp: 2021-04-08T07:57:29Z
              cni.projectcalico.org/podIP: 10.42.0.189/32
              cni.projectcalico.org/podIPs: 10.42.0.189/32
              field.cattle.io/ports:
                [[{"containerPort":80,"dnsName":"nginx-ingress-controller","kind":"ClusterIP","name":"http","protocol":"TCP"},{"containerPort":443,"dnsNam...
              kubernetes.io/psp: nginx-ingress
Status:       Running
IP:           10.42.0.189
IPs:
  IP:           10.42.0.189
Controlled By:  ReplicaSet/nginx-ingress-controller-6d6596d45b
Containers:
  nginx-ingress-controller:
    Container ID:  docker://4a65248345582e3f6991e71f6d22d927fce06f0cb65b6165a020e417dca3f644
    Image:         nexus3/air-gap/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0
    Image ID:      docker-pullable://nexus3/air-gap/kubernetes-ingress-controller/nginx-ingress-controller@sha256:bc95fdbd64452185e73599dc4d69e5346b307c8fdd0851149a62886930f5a4d2
    Ports:         80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=nginx-ingress/nginx-ingress-default-backend
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=nginx-ingress/nginx-ingress-controller
    State:          Running
      Started:      Sun, 11 Apr 2021 22:45:13 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sun, 11 Apr 2021 22:45:01 +0000
      Finished:     Sun, 11 Apr 2021 22:45:02 +0000
    Ready:          True
    Restart Count:  4
    Limits:
      cpu:     700m
      memory:  768Mi
    Requests:
      cpu:      300m
      memory:   500Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-6d6596d45b-49mzk (v1:metadata.name)
      POD_NAMESPACE:  nginx-ingress (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-token-ddmtf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  nginx-ingress-token-ddmtf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-token-ddmtf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
longwuyuan commented 3 years ago

if you don't want to use helm cli and prefer to use the Rancher Catalog, I think you should close this issue and create a issue in Rancher's repo

/remove-kind support /triage needs-information

rsitavares commented 3 years ago

I'm having the same issue. Using helm:

NAME                            NAMESPACE       REVISION    UPDATED                                 STATUS      CHART                               APP VERSION
aws-load-balancer-controller    kube-system     9           2021-07-16 15:56:40.753045 -0300 -03    deployed    aws-load-balancer-controller-1.2.3  v2.2.1
ingress-nginx                   ingress-nginx   2           2021-07-16 16:51:03.306723 -0300 -03    deployed    ingress-nginx-3.34.0                0.47.0

When I change the target group health check port to 80 (HTTP) the error logs stop. Whenever I switch it back to the traffic port 443 (HTTPS) the error logs resume. The application traffic, however, works fine.

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

balonik commented 2 years ago

You can't use use-proxy-protocol: "true" with AWS NLB. It must be on the default false

hd-deman commented 2 years ago

@balonik why?

balonik commented 2 years ago

@hd-deman nevermind, I missed that externalTrafficPolicy is set to Cluster instead of Local.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/6996#issuecomment-1000883956): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.