search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.76k stars 5.42k forks source link

Updating SSH known hosts via ConfigMap edit doesn't work properly #5794

Open rbkaspr opened 3 years ago

rbkaspr commented 3 years ago

Checklist:

Describe the bug

Recently the team hosting our corporate Gitlab instance had a bit of a mix-up on their end the caused a new host to join the loadbalancer group of Gitlab hosting servers, which resulted in a new ECDSA fingerprint causing issues when Argo tried to connect. Our ArgoCD instance immediately started refusing to talk to Gitlab due to the key mismatch, so we attempted to add the fingerprint via the argocd-ssh-known-hosts-cm ConfigMap. Despite those changes and multiple bounces of the ArgoCD pods, Argo continued to reject connections to Gitlab, even though we could look at the container filesystem and see that the mounted knownhosts file had been updated.

The only solution to to problem that we could find was to delete all existing host keys via the UI, and then re-add them via the UI by pasting the latest ssh-keyscan output

To Reproduce

  1. Setup Argo pointing at a hosted Git server (something like Gitea should probably be fine for testing.
  2. Have the server regenerate its host keys
  3. Watch ArgoCD panic
  4. Try adding a new hostkey entry via ConfigMap
  5. Watch Argo continue to panic

Expected behavior

Considering that most of the Argo pods are largely stateless and get their configuration from Kubernetes objects, I would expect the adding an offending host key to the corresponding ConfigMap and bouncing the ArgoCD pods would cause the key to be loaded and accepted. For whatever reason that doesn't appear to be the case and the UI seems to be the only way to get around this issue, which doesn't square well with ArgoCD's declarative/Gitops design ethos.

Version

v1.8.7+eb3d1fb
adetalhouet commented 2 years ago

I'm also observing that updating the ArgoCD CR directly with initialSSHKnownHosts doesn't do anything. Once the section is added to the CR, I have to delete the CM for the additional data to be picked up.