VannTen
commented
9 months ago Because this ArgoCD addon is tricky to upgrade and Kubespray includes it as a supported component, I believe one of the following needs to be done:
Either: remove support for the ArgoCD addon and let people deploy it as they see fit and worry about upgrading by themselves. At least re-running
cluster.ymlwould not cause data loss and it won't be Kubespray's problem anymore.
Or: add Kubespray documentation related to how the ArgoCD addon is to be used. I couldn't find any documentation about this addon in Kubespray yet. Perhaps one should enable the ArgoCD addon once, install via Kubespray and then disable it to avoid data loss? To upgrade to a new version in the future, perhaps one should be instructed to first create an ArgoCD backup as described in Disaster Recovery, then re-enable the addon, re-run the playbook, import the ArgoCD backup data and finally disable the addon again to avoid accidental data loss.
Or: Kubespray could be made to do all this automatically. If it could do so, there's value-add in having the Kubespray playbook install ArgoCD as opposed to installing it the regular way. Implementation ideas follow. During addon installation, addon manifests (
*.yaml) are dumped to the first control plane node (/etc/kubernetes/addons/..). This node already has akubeconfigfile with admin access. The playbook may detect the existence of ArgoCD, perform backup procedures by running containers as described in Disaster Recovery, then run ArgoCD's all-in-oneinstall.yamllike it does now, then restore from backup and finally clean up the local backups.
I personally would go with 1, in the direction of #6400 . Maybe server apply could help us in this case, but that's not available in the custom kube.py module of kubespray.
@floryut @yankay : I propose the following: let's give us some delay to see if anyone come forward to support properly the ArgoCD addon, if not, delete it. (before 2.25 at the latest).
Wdyt ?
k8s-triage-robot
vaibhav2107
k8s-ci-robot
What happened?
After re-running
cluster.yml(with or without--tags=argocd), ArgoCD gets reinstalled each time from the upstream all-in-oneinstall.yamlfile:install.yamlcontains various resource definitions, among which severalConfigMapresources. Some of these (e.g.argocd-ssh-known-hosts-cm,argocd-cm) are used for storing applications state which the ArgoCD administrator may wish to change.argocd-ssh-known-hosts-cmis certainly used by people who wish to connect ArgoCD to their own private git repositories running on a self-hosted server.argocd-cmmay be adjusted by people with more special Ingress, to work around Why is my application stuck in Progressing state?, as described here and here.Re-running Kubespray having the side-effect of rebuilding these
ConfigMapobjects and losing data is unexpected.What did you expect to happen?
This may be a bit of an upstream documentation problem as well. The Upgrading guide for ArgoCD does not state that one is supposed to backup the ArgoCD settings each time versions are switched (or the same version is re-applied). It hints that during major version upgrades it's recommended to take a backup as described on the Disaster Recovery documentation. It's not explicitly clear that
ConfigMap-stored settings will always be lost when runninginstall.yaml, regardless of whether you're upgrading at all or reinstalling the same version.It seems like the proper way to upgrade ArgoCD is to always:
install.yamlfor the new versionUnless I'm misunderstanding something, I believe the ArgoCD Upgrading/Overview documentation should be updated to reflect this.
Nevertheless, given that Kubespray supports an ArgoCD addon, it also becomes Kubespray's problem how this addon is managed.
Because this ArgoCD addon is tricky to upgrade and Kubespray includes it as a supported component, I believe one of the following needs to be done:
cluster.ymlwould not cause data loss and it won't be Kubespray's problem anymore.*.yaml) are dumped to the first control plane node (/etc/kubernetes/addons/..). This node already has akubeconfigfile with admin access. The playbook may detect the existence of ArgoCD, perform backup procedures by running containers as described in Disaster Recovery, then run ArgoCD's all-in-oneinstall.yamllike it does now, then restore from backup and finally clean up the local backups.How can we reproduce it (as minimally and precisely as possible)?
argocd_enabled: true+argocd_admin_password: ...)cluster.yml(orcluster.yml --tags=argocd) to get it installed1234:kubectl port-forward -n argocd service/argocd-server 1234:80adminand the password specified inargocd_admin_passwordargocd-ssh-known-hosts-cmConfigMap:kubectl -n argocd describe cm argocd-ssh-known-hosts-cm | grep example.comcluster.yml(orcluster.yml --tags=argocd) once again. We have not changed the ArgoCD version in the inventory or done any other changes.argocd-ssh-known-hosts-cmConfigMap) got reset to their default initial values as seen in the upstreaminstall.yamlall-in-one file.kubectl -n argocd describe cm argocd-ssh-known-hosts-cm | grep example.comExample SSH host key that may be added during this test:
This example SSH host key is actually one of the
github.comSSH host keys. We have to use some vaid base64-encoded data for ArgoCD to accept it, so using something real seemed like the best choice.OS
Irrelevant
Version of Ansible
I'm running the container image for the
v2.24release (quay.io/kubespray/kubespray:v2.24):Version of Python
Python 3.10.12
Version of Kubespray (commit)
64447e745e53d3f486356d03ecb195729a302ea1
Network plugin used
calico
Full inventory with variables
No response
Command used to invoke ansible
No response
Output of ansible run
No response
Anything else we need to know
No response