search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.93k stars 1.46k forks source link

Ingress validator failing for ingress rules without http path #3158

Closed mohammed-infstones closed 1 year ago

mohammed-infstones commented 1 year ago

Describe the bug A concise description of what the bug is. When deleting or patching the ingress controller, its stuck in terminating state with the following error message 


This was deployed using a helm chart. Deleted the helm chart but no use.Also, do not see the ingress resources anymore in AWS. There are other ingress controllers with no issues.

## Error from aws lb controller

{"level":"error","ts":"2023-04-17T17:37:23Z","msg":"Reconciler error","controller":"ingress","object":{"name":"xxxxx"},"namespace":"","name":"xxxxx","reconcileID":"044e11cb-faca-4bb2-a476-31e519ad4d64","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": failed to call webhook: Post \"https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1-ingress?timeout=10s\": EOF"}
{"level":"info","ts":"2023-04-17T17:38:04Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"xxxxx\",\"resources\":{}}"}
{"level":"info","ts":"2023-04-17T17:38:05Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"xxxxx"}
{"level":"info","ts":"2023-04-17T17:38:05Z","logger":"backend-sg-provider","msg":"No ingress found, backend SG can be deleted","SG ID":""}
{"level":"error","ts":"2023-04-17T17:38:05Z","msg":"Reconciler error","controller":"ingress","object":{"name":"xxxxx"},"namespace":"","name":"xxxxx","reconcileID":"d6dffca7-9660-4ea6-8586-fc126cc643ac","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": failed to call webhook: Post \"https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1-ingress?timeout=10s\": EOF"}
{"level":"info","ts":"2023-04-17T17:39:27Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"xxxxx\",\"resources\":{}}"}
{"level":"info","ts":"2023-04-17T17:39:28Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"xxxxx"}
{"level":"info","ts":"2023-04-17T17:39:28Z","logger":"backend-sg-provider","msg":"No ingress found, backend SG can be deleted","SG ID":""}
2023/04/17 17:39:28 http: panic serving x.x.x.x:54124: runtime error: invalid memory address or nil pointer dereference
goroutine 2178 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1854 +0xbf
panic({0x7fe00ba2afa0, 0x7fe00d10b3f0})
    /usr/local/go/src/runtime/panic.go:890 +0x263
sigs.k8s.io/aws-load-balancer-controller/webhooks/networking.(*ingressValidator).checkIngressAnnotationConditions(0xc0006565f0, 0xc000f53a20)
    /workspace/webhooks/networking/ingress_validator.go:177 +0x6d
sigs.k8s.io/aws-load-balancer-controller/webhooks/networking.(*ingressValidator).ValidateUpdate(0x7fe00be2f338?, {0x7fe00be2f3e0, 0xc000ff8750}, {0x7fe00be1db28?, 0xc000f53a20}, {0x7fe00be1db28?, 0xc000f53b80})
    /workspace/webhooks/networking/ingress_validator.go:81 +0xca
sigs.k8s.io/aws-load-balancer-controller/pkg/webhook.(*validatingHandler).handleUpdate(_, {_, _}, {{{0xc000c4b710, 0x24}, {{0xc000d0fe00, 0x11}, {0xc000d99f10, 0x2}, {0xc000d99f12, ...}}, ...}})
    /workspace/pkg/webhook/validating_handler.go:74 +0x52d
sigs.k8s.io/aws-load-balancer-controller/pkg/webhook.(*validatingHandler).Handle(_, {_, _}, {{{0xc000c4b710, 0x24}, {{0xc000d0fe00, 0x11}, {0xc000d99f10, 0x2}, {0xc000d99f12, ...}}, ...}})
    /workspace/pkg/webhook/validating_handler.go:34 +0x310
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_, {_, _}, {{{0xc000c4b710, 0x24}, {{0xc000d0fe00, 0x11}, {0xc000d99f10, 0x2}, {0xc000d99f12, ...}}, ...}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/webhook/admission/webhook.go:169 +0xfd
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc0005a0a40, {0x7fdfdd4c3be0?, 0xc0001ba0f0}, 0xc000f56100)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/webhook/admission/http.go:98 +0xeb5
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7fdfdd4c3be0, 0xc0001ba0f0}, 0x7fe00be2e100?)
    /go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:60 +0xd4
net/http.HandlerFunc.ServeHTTP(0x7fe00be2e1c0?, {0x7fdfdd4c3be0?, 0xc0001ba0f0?}, 0x7fe0099d2e80?)
    /usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7fe00be2e1c0?, 0xc0005f2620?}, 0xc000f56100)
    /go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xb8
net/http.HandlerFunc.ServeHTTP(0xc0005f2620?, {0x7fe00be2e1c0?, 0xc0005f2620?}, 0xc000f58140?)
    /usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x7fe00be2e1c0, 0xc0005f2620}, 0xc000f56100)
    /go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:108 +0xbf
net/http.HandlerFunc.ServeHTTP(0xc0005f2620?, {0x7fe00be2e1c0?, 0xc0005f2620?}, 0x7fe00b1d34f3?)
    /usr/local/go/src/net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc000f58125?, {0x7fe00be2e1c0, 0xc0005f2620}, 0xc000f56100)
    /usr/local/go/src/net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0x7fe00be1f798?}, {0x7fe00be2e1c0, 0xc0005f2620}, 0xc000f56100)
    /usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc000c04fc0, {0x7fe00be2f3e0, 0xc000950990})
    /usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3089 +0x5ed
{"level":"error","ts":"2023-04-17T17:39:28Z","msg":"Reconciler error","controller":"ingress","object":{"name":"xxxxx"},"namespace":"","name":"xxxxx","reconcileID":"d7ca05fe-2f2b-4d46-b13e-9dedd00f1ff1","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": failed to call webhook: Post \"https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1-ingress?timeout=10s\": EOF"}
{"level":"info","ts":"2023-04-17T17:42:12Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"xxxxx\",\"resources\":{}}"}
{"level":"info","ts":"2023-04-17T17:42:13Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"xxxxx"}
{"level":"info","ts":"2023-04-17T17:42:13Z","logger":"backend-sg-provider","msg":"No ingress found, backend SG can be deleted","SG ID":""}
{"level":"error","ts":"2023-04-17T17:42:13Z","msg":"Reconciler error","controller":"ingress","object":{"name":"xxxxx"},"namespace":"","name":"xxxxx","reconcileID":"bfcec579-35ab-4313-8f5b-56569eaa23b3","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": failed to call webhook: Post \"https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1-ingress?timeout=10s\": EOF"}
{"level":"info","ts":"2023-04-17T17:47:40Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"xxxxx\",\"resources\":{}}"}
{"level":"info","ts":"2023-04-17T17:47:42Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"xxxxx"}
{"level":"info","ts":"2023-04-17T17:47:42Z","logger":"backend-sg-provider","msg":"No ingress found, backend SG can be deleted","SG ID":""}

Steps to reproduce

Deploy ingress using a helm chart and update or terminate it

Expected outcome A concise description of what you expected to happen. The ingress should be updated or deleted

Environment Dev EKS

Additional Context:

kishorj commented 1 year ago

@mohammed-infstones, would you be able to share the ingress spec?

mohammed-infstones commented 1 year ago 
kind: Ingress
metadata:
  name: {{ .Values.appName }}-server
  namespace: {{ .Values.k8sNamespace }}
  labels:
    aws-load-balancer-scheme: "{{ .Values.lbScheme }}"
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: {{ .Values.appName }}
    external-dns.alpha.kubernetes.io/hostname: admin.cloud.{{ .Values.domainName }}
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.cloudSslCertArn }}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":{{ .Values.adminTargetPort }}}]'
    alb.ingress.kubernetes.io/load-balancer-name: wsa-alb-{{ .Values.domainNameTransposed }}
    alb.ingress.kubernetes.io/scheme: "{{ .Values.lbScheme }}"
    alb.ingress.kubernetes.io/target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: "{{ .Values.lbScheme }}"
spec:
  rules:
    - http:
    - http:
        paths:
          - pathType: ImplementationSpecific
            backend:
              service:
                 name: {{ .Values.appName }}-server
                 port:
                   number: {{ .Values.adminTargetPort }}
mohammed-infstones commented 1 year ago 

kind: Service
metadata:
  namespace: {{ .Values.k8sNamespace }}
  name: {{ .Values.appName }}-server
spec:
  ports:
    - port: {{ .Values.adminTargetPort }}
      targetPort: {{ .Values.adminTargetPort }}
      protocol: TCP
  type: NodePort
  selector:
    app: {{ .Values.appName }}```
kishorj commented 1 year ago

/kind bug

mohammed-infstones commented 1 year ago

When I try to patch it by removing the finalizers, I get the error {"level":"error","ts":"2023-04-17T17:42:13Z","msg":"Reconciler error","controller":"ingress","object":{"name":"webserviceadmin"},"namespace":"","name":"webserviceadmin","reconcileID":"bfcec579-35ab-4313-8f5b-56569eaa23b3","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": failed to call webhook: Post \"https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1-ingress?timeout=10s\": EOF"}

kishorj commented 1 year ago

@mohammed-infstones, it is due to a bug in the ingress validator, introduced in v2.5.0 #2735. The validator is currently not able to handle ingress rules with empty http path. We will fix it as soon as possible in a patch release.

mohammed-infstones commented 1 year ago

@kishorj will it fix if I rollback to previous version?

kishorj commented 1 year ago

@mohammed-infstones, previous version 2.4.x is not affected by this bug. We will release v2.5.1 with the fix.

mohammed-infstones commented 1 year ago

@kishorj thank you! Any timeline for that fix?

kishorj commented 1 year ago

@kishorj thank you! Any timeline for that fix?

asap

kishorj commented 1 year ago

PR #3160