search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.33k stars 8.22k forks source link

upstream prematurely closed connection while reading response header from upstream #9877

Closed doramar97 closed 3 weeks ago

doramar97 commented 1 year ago

What happened: This is the error i'm getting, I removed sensitive information. A client try to access our api and got this errors.

3.18.201.127 - - [17/Apr/2023:20:24:55 +0000] "POST /api/authentication/token HTTP/1.1" 502 150 "-" "axios/0.23.0" 340 0.016 [XXX] [] XXX 0 0.000 502 baef50777d0f3651f32233c609ad0e7e

2023/04/17 20:24:55 [error] 397#397: *56382572 upstream prematurely closed connection while reading response header from upstream, client: XXX server: XXX, request: "POST /api/authentication/token HTTP/1.1", upstream: "http://XXX/api/authentication/token", host: "XXX" What you expected to happen:

I expect the api request to go to our internal services, instead of get 502 error. (all our services were up and running).

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Kubernetes version (use kubectl version): Client Version: v1.26.4 Kustomize Version: v4.5.7 Server Version: v1.21.14-eks-48e63af

Environment:

USER-SUPPLIED VALUES:
controller:
  config:
    use-forwarded-headers: "true"
    use-proxy-protocol: "true"
  metrics:
    enabled: "true"
  replicaCount: 2
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: benga-trail-logs
      service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: ELBLogs/production
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    externalTrafficPolicy: Local
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=1.1.0
              helm.sh/chart=ingress-nginx-4.0.12
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress
Controller:   k8s.io/ingress-nginx
Events:       <none>
Name:             ingress-nginx-controller-55cf99bb45-k2xkv
Namespace:        ingress
Priority:         0
Service Account:  ingress-nginx
Node:             ip-10-10-39-97.ec2.internal/10.10.39.97
Start Time:       Sun, 29 Jan 2023 15:31:04 +0200
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  pod-template-hash=55cf99bb45
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.10.43.67
IPs:
  IP:           10.10.43.67
Controlled By:  ReplicaSet/ingress-nginx-controller-55cf99bb45
Containers:
  controller:
    Container ID:  containerd://b17a502cd09674a7c2a5e86cb95d672a573b6d9b362e873373a895ccda69fb7a
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    Image ID:      k8s.gcr.io/ingress-nginx/controller@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 29 Jan 2023 15:31:14 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-55cf99bb45-k2xkv (v1:metadata.name)
      POD_NAMESPACE:  ingress (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-574bp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-574bp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

I'm Running with the same configuration in my staging and dev environments, and i'm not seeing similar issues there. There was no downtime or a any unusual traffic on our services. Thanks.

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

@doramar97 there are other questions asked in the new issue template that relate to the ingress object and the request sent to the LB etc etc.

Without all that info, this issue will just track your error message as there is no context related to configuration, or the request sent/received etc etc

You can look at the new issue template and answer those questions for readers to get an idea of the configuration, the live state and the transactions.

longwuyuan commented 1 year ago

/remove-kind bug

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

longwuyuan commented 3 weeks ago

/kind support /remove-kind feature

There is no describe of the ingress resource or the controller service or the curl command in full detail or the complete logs of the controller or the state of the backend etc etc. So there is nothing to analyze or reproduce or comment on. Upstream connections can break due to networking or load or broken process in upstream. Without the above mentioned details there is nothing that the reader here can respond with helpfully.

Since there is no action item here and since there is acute shortage of resources to keep issues open without action items, I am closing this issue.

/close

k8s-ci-robot commented 3 weeks ago

@longwuyuan: Those labels are not set on the issue: kind/feature

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9877#issuecomment-2345260270): >/kind support >/remove-kind feature > >There is no describe of the ingress resource or the controller service or the curl command in full detail or the complete logs of the controller or the state of the backend etc etc. So there is nothing to analyze or reproduce or comment on. Upstream connections can break due to networking or load or broken process in upstream. Without the above mentioned details there is nothing that the reader here can respond with helpfully. > >Since there is no action item here and since there is acute shortage of resources to keep issues open without action items, I am closing this issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
k8s-ci-robot commented 3 weeks ago

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9877#issuecomment-2345260270): >/kind support >/remove-kind feature > >There is no describe of the ingress resource or the controller service or the curl command in full detail or the complete logs of the controller or the state of the backend etc etc. So there is nothing to analyze or reproduce or comment on. Upstream connections can break due to networking or load or broken process in upstream. Without the above mentioned details there is nothing that the reader here can respond with helpfully. > >Since there is no action item here and since there is acute shortage of resources to keep issues open without action items, I am closing this issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.