ingress controller is creating HTTP2 targetgroups when my healthchecks only pass on HTTP1

[meyerkev]

Describe the bug Our healthchecks only pass when using HTTP1 and our Target group is always HTTP2 even when we set it explicitly.

Specifically, we're running thanos-query as part of the thanos chart so we don't control their compatibility.

Steps to reproduce

1. Create an ingress with this particular set of annotations

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
   annotations:
   alb.ingress.kubernetes.io/backend-protocol: HTTP
   alb.ingress.kubernetes.io/backend-protocol-version: HTTP1

2. Apply the ingress

3. Get these results

Expected outcome I should get an HTTP/1 target group that checks, what, in my case, was port 9090 on / every 5 seconds.

Instead, I get an HTTP/2 target group instead.

Environment

• AWS Load Balancer controller version -> v2.2.0, upgraded to v2.4.0 and then the same thing happens.
• Kubernetes version -> 1.20 (Yes, I know. We're working on it)
• Using EKS (yes/no), if so version? Yes, eks.12

Additional Context: I did once manage to get it to work and make me an HTTP/1 version and it did in fact briefly work.

The full ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP1
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:795586375822:certificate/83453e70-53c2-4e5f-be34-e55b15b3af22
    alb.ingress.kubernetes.io/group.name: usw2-staging-internal
    alb.ingress.kubernetes.io/group.order: "106"
    alb.ingress.kubernetes.io/healthcheck-port: "9090"
    alb.ingress.kubernetes.io/inbound-cidrs: 10.0.0.0/8
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200,302,301,304
    alb.ingress.kubernetes.io/target-type: instance
    external-dns.alpha.kubernetes.io/alias: "true"
    external-dns.alpha.kubernetes.io/hostname: thanos-query.usw2-staging.mywebsite.io
    external-dns.alpha.kubernetes.io/ttl: "60"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/backend-protocol":"HTTP","alb.ingress.kubernetes.io/backend-protocol-version":"HTTP1","alb.ingress.kubernetes.io/certificate-arn":"arn:aws:acm:us-west-2:795586375822:certificate/83453e70-53c2-4e5f-be34-e55b15b3af22","alb.ingress.kubernetes.io/group.name":"usw2-staging-internal","alb.ingress.kubernetes.io/group.order":"106","alb.ingress.kubernetes.io/healthcheck-port":"9090","alb.ingress.kubernetes.io/inbound-cidrs":"10.0.0.0/8","alb.ingress.kubernetes.io/listen-ports":"[{\"HTTP\": 80}, {\"HTTPS\":443}]","alb.ingress.kubernetes.io/scheme":"internal","alb.ingress.kubernetes.io/ssl-redirect":"443","alb.ingress.kubernetes.io/success-codes":"200,302,301","alb.ingress.kubernetes.io/target-type":"instance","external-dns.alpha.kubernetes.io/alias":"true","external-dns.alpha.kubernetes.io/hostname":"thanos-query.usw2-staging.mywebsite.io","external-dns.alpha.kubernetes.io/ttl":"60","kubernetes.io/ingress.class":"alb"},"name":"thanos-query","namespace":"monitoring"},"spec":{"rules":[{"host":"thanos-query.usw2-staging.mywebsite.io","http":{"paths":[{"backend":{"service":{"name":"thanos-query","port":{"number":10901}}},"pathType":"ImplementationSpecific"}]}}]}}
    kubernetes.io/ingress.class: alb
  creationTimestamp: "2023-04-18T00:36:56Z"
  finalizers:
  - group.ingress.k8s.aws/usw2-staging-internal
  generation: 1
  name: thanos-query
  namespace: monitoring
  resourceVersion: "159293350"
  uid: 8e14979d-d0d4-4af6-a1c6-695ed1bd47dd
spec:
  rules:
  - host: thanos-query.usw2-staging.mywebsite.io
    http:
      paths:
      - backend:
          service:
            name: thanos-query
            port:
              number: 10901
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: internal-k8s-usw2stagingintern-84b72ea72c-450260535.us-west-2.elb.amazonaws.com

[meyerkev]

Never mind me, our service was overriding the ingress with HTTP2