ApplicationSet Webhook fails health checks from ingress

[cristicalin]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I'm trying to deploy the argocd application set controller and expose the controller web hook through AWS Loadbalancer controller (which uses an ALB in front). Since the ALB does it's own health checks the resulting ALB target group never gets registered because the web hook handler responds with HTTP 400 to anything that is not a valid web hook call.

To Reproduce

• Deploy any health checking load balancer controller (such as AWS LB controller) on an EKS cluster.
• deploy argocd using the helm chart with following configuration for the application set:

applicationSet:
  enabled: true

  ingress:
    enabled: true

    ingressClassName: alb

    hostname: appset-argocd.example.com

    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/backend-protocol: HTTPS
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
      alb.ingress.kubernetes.io/certificate-arn: <some-valid-arn-for-example-com>
      alb.ingress.kubernetes.io/group.name: my-group-name
      kubernetes.io/ingress.class: alb
      external-dns.alpha.kubernetes.io/set-identifier: my-eks
      external-dns.alpha.kubernetes.io/aws-weight: 100

The ALB target group remains unhealthy.

curl commands from a port-forwarded webhook:

± curl localhost:7000/api/webhook -v
* Host localhost:7000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7000...
* Connected to localhost (::1) port 7000
> GET /api/webhook HTTP/1.1
> Host: localhost:7000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 01 Apr 2024 14:23:48 GMT
< Content-Length: 22
<
Unknown webhook event
* Connection #0 to host localhost left intact

 ± curl localhost:7000/api/webhook -vI
* Host localhost:7000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7000...
* Connected to localhost (::1) port 7000
> HEAD /api/webhook HTTP/1.1
> Host: localhost:7000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Date: Mon, 01 Apr 2024 14:24:00 GMT
Date: Mon, 01 Apr 2024 14:24:00 GMT
< Content-Length: 22
Content-Length: 22
<

* Connection #0 to host localhost left intact

± curl localhost:7000/ -vI
* Host localhost:7000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7000...
* Connected to localhost (::1) port 7000
> HEAD / HTTP/1.1
> Host: localhost:7000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Date: Mon, 01 Apr 2024 14:34:09 GMT
Date: Mon, 01 Apr 2024 14:34:09 GMT
< Content-Length: 19
Content-Length: 19
<

* Connection #0 to host localhost left intact

Expected behavior

The root (/) of the web hook should have a valid handler that returns a fake HTTP 200 for health checks so that health checking loadbalancers can detect the backend server health.

Screenshots

Version

± argocd version
argocd: v2.10.5+335875d.dirty
  BuildDate: 2024-03-28T18:33:38Z
  GitCommit: 335875d13e018bed6e03873f4742582582964745
  GitTreeState: dirty
  GoVersion: go1.22.1
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.10.5+335875d.dirty
  BuildDate: 2024-03-28T18:33:38Z
  GitCommit: 335875d13e018bed6e03873f4742582582964745
  GitTreeState: dirty
  GoVersion: go1.22.1
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs

Application Set Controller doesn't seem to output any logs.

[kaessert]

We're facing the same

[pniebylski-zilch]

Same here, target groups health check fails

[dmarquez-splunk]

Also seeing this issue. Anyone find a way to configuring the health check into thinking the service is healthy? Big blocker for us right now

[mheiges]

a workaround,

applicationSet:
...
  ingress:
...
    annotations:
      alb.ingress.kubernetes.io/healthcheck-path: /api/webhook
      alb.ingress.kubernetes.io/success-codes: 400
...

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and tell us if the issue is still present, please?