Random Webhook token verification failure (Gitlab)

[c-buisson]

Describe the bug

I am experiencing this bug where ArgoCD server will randomly fail to verify the Gitlab webhook token. The webhook is working but sometimes ArgoCD will not be able to verify the token. After a few minutes, ArgoCD will be able to verify the token again (nothing was updated/changed on either Gitlab or ArgoCD).

To Reproduce

I followed this page to setup the Webhook and configure ArgoCD.

Expected behavior

ArgoCD should always be able to validate the webhook token.

Screenshots

Version

argocd: v2.8.0+804d4b8
  BuildDate: 2023-08-07T19:41:16Z
  GitCommit: 804d4b8ca6bc4c2cf02c5c971aa923ec5b8623f0
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.8.0+804d4b8
  BuildDate: 2023-08-07T14:25:33Z
  GitCommit: 804d4b8ca6bc4c2cf02c5c971aa923ec5b8623f0
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.1.0 2023-06-19T16:58:18Z
  Helm Version: v3.12.1+gf32a527
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

Logs

time="2023-08-30T13:38:34Z" level=info msg="GitLab webhook token verification failed" security=3
time="2023-08-30T13:38:34Z" level=info msg="Webhook processing failed: X-Gitlab-Token validation failed"

[Kamaradeivanov]

Same issue here with argocd v2.8.3

[aydinseven7]

Same issue here with argocd v2.6.7

Edit: We have added the Gitlab Webhook token through a secret that is referenced in the values.yaml. However, ArgoCD also saves a Gitlab Webhook Secret in the argocd-secret. Once I deleted that, the webhook went through without problems.

[daquan]

I also encountered this problem, argocd: version 2.9.0, sometimes normal and sometimes status code 400。

[SergeyLadutko]

Same problem

[alex0403lin]

Same problem, I restarted deployment/argocd-server. After restarting, the webhook works normally.

[sarath-ig]

This is occurring always for me in v2.9.3 as well. I've tried restarting the argocd-server deployment as well, but seems to no avail.

Does anybody have any pointers as to how to debug/fix this?

Any help is much appreciated. Thanks

[jibill]

same problem: v2.9.3

[mrkwtz]

Hey @sarath-ig @jibill

We ran into the same problem when we upgraded from 2.8.0 to 2.9.3 and it turned out we had set a token in the GitLab webhook configuration but not in the argocd-secret. It seems like in former versions (at least 2.8.0 and below) ArgoCD ignored the X-Gitlab-Token if you had set no secret in ArgoCD and therefore it worked.

After the update it seems like ArgoCD is not so lenient anymore and if you have set no secret in ArgoCD but did provide a secret via X-Gitlab-Token it fails (although I'm not able to find the respective code change in ArgoCD code or code of the utilized library https://github.com/go-playground/webhooks).

So to resolve the problem make sure that if you have set a secret token in the GitLab Webhook confiuration you also have set the same in the argocd-secret secret.

[MichaelTrip]

I can confirm i have this issue too. It didn´t occur with 2.8.x. But with the release 2.9.3 sometimes the webhook doesn´t work.

[EM-Savard]

Hi guys,

I have just upgraded to Argo CD 2.9.6, and at the beginning I was still getting the X-Gitlab-Token validation failed error. After restarting the argo-server instances (@alex0403lin), the error seams to be gone! Before the reboot, I was getting the error more than 50% of the time and after the reboot, all the webhooks (40+) have been successfully sent to Argo. So I'll keep track of the success rate and keep this thread updated if any issues occurs with the Gitlab webhook!

[sudoexec]

Only 2 of 36 webhooks succeed in version 2.10.6.