Ingress configuration - Failed health checks

[abalestrini]

Describe the bug

I got argoCD implemented in my EKS cluster and wanted to expose the UI through the ingress that this Helm chart provides by using an AWS ALB. The fact is that while the ingress is created properly I got a health check error in the ALB TG. The configuration is the following

     configs:
        params:
          server.insecure: true

      global:
        domain: argo.staging.my.domain.com

      server:
        ingress:
          enabled: true
          controller: aws
          ingressClassName: alb
          annotations:
            alb.ingress.kubernetes.io/load-balancer-name: 'k8s-Argo-CD-alb'
            alb.ingress.kubernetes.io/scheme: internal
            alb.ingress.kubernetes.io/target-type: ip
            alb.ingress.kubernetes.io/subnets: subnet-xxxxxxxxx, subnet-xxxxxxxxx
            alb.ingress.kubernetes.io/backend-protocol: HTTP
            alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
            alb.ingress.kubernetes.io/ssl-redirect: '443'

          aws:
            serviceType: ClusterIP
            backendProtocolVersion: GRPC

Here you can see what the ALB target groups look like

Related helm chart

argo-cd

Helm chart version

6.7.1

To Reproduce

-

Expected behavior

The ingress should be created correctly allowing me to use it to reach the argocd UI.

Screenshots

Additional context

Sharing the ingress manifest as it was created by the chart:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/conditions.argo-cd-argocd-server-grpc: |
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/load-balancer-name: k8s-staging-Argo-CD-alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/subnets: subnet-xxxxxxxxx, subnet-xxxxxxxxx
    alb.ingress.kubernetes.io/target-type: ip
    meta.helm.sh/release-name: argo-cd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2024-10-14T16:18:15Z"
  finalizers:
  - ingress.k8s.aws/resources
  generation: 1
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.10.2
    helm.sh/chart: argo-cd-6.7.1
  name: argo-cd-argocd-server
  namespace: argocd
  resourceVersion: "86178195"
  uid: 6622bfd8-c7b7-47e2-9245-9c22bee5d42b
spec:
  ingressClassName: alb
  rules:
  - host: argo.staging.my.domain.com
    http:
      paths:
      - backend:
          service:
            name: argo-cd-argocd-server-grpc
            port:
              number: 80
        path: /
        pathType: Prefix
      - backend:
          service:
            name: argo-cd-argocd-server
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer: {}

and here is the backend service created by the helm chart too

apiVersion: v1
kind: Service
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol-version: GRPC
    meta.helm.sh/release-name: argo-cd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2024-10-14T16:18:10Z"
  labels:
    app.kubernetes.io/component: server-gprc
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-server-grpc
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.10.2
    helm.sh/chart: argo-cd-6.7.1
  name: argo-cd-argocd-server-grpc
  namespace: argocd
  resourceVersion: "86178111"
  uid: 3b5005e2-b74d-44db-8909-c5e8911f4ffb
spec:
  clusterIP: 10.100.26.122
  clusterIPs:
  - 10.100.26.122
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/name: argocd-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

[timam]

I used HTTPS as backend protocol. My target group looks healthy.

      kubernetes.io/ingress.class : alb
      alb.ingress.kubernetes.io/backend-protocol: HTTPS
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip

[abalestrini]

Hi @timam I think that using the backend-protocol as HTTPS would conflict with my current config because I set up the server with insecure: true. I might give it a try though

[abalestrini]

Thanks @timam for your comment, it worked. Sharing the final configuration

      configs:
        params:
          server.insecure: false

      global:
        domain: argo.staging.my.domain.com

      server:
        ingress:
          enabled: true
          controller: aws
          ingressClassName: alb
          annotations:
            alb.ingress.kubernetes.io/load-balancer-name: 'k8s-staging-argo-cd-alb'
            alb.ingress.kubernetes.io/scheme: internal
            alb.ingress.kubernetes.io/target-type: ip
            alb.ingress.kubernetes.io/subnets: subnet-xxxxxxxxx, subnet-xxxxxxxxx
            alb.ingress.kubernetes.io/backend-protocol: HTTPS
            alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
            alb.ingress.kubernetes.io/ssl-redirect: '443'

          aws:
            serviceType: ClusterIP
            backendProtocolVersion: GRPC