Incomplete upgrading documentation

jfdumont commented 4 years ago

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

[x ] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
[ x] I've included steps to reproduce the bug.
[ x] I've pasted the output of argocd version.

Describe the bug I upgrade from v1.5.2 to v1.5.3. as describe in upgrading documentation

The patch release does not introduce any breaking changes. 
So if you are upgrading from v1.5.1 to v1.5.3 there should be no special instructions to follow.

no special instructions so I apply manifests. But both argocd-rbac-cm and argocd-cm have been reset.

Expected behavior

expected : warning that resources can or will be reset.

Version

argocd: v1.5.3+095c5d6
  BuildDate: 2020-05-02T04:19:50Z
  GitCommit: 095c5d616b1cb39f87e8f7a54cabea2643e1c99a
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.5.3+095c5d6
  BuildDate: 2020-05-02T04:21:47Z
  GitCommit: 095c5d616b1cb39f87e8f7a54cabea2643e1c99a
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

jannfis commented 4 years ago

Hi @jfdumont,

applying the manifests should usually not overwrite your existing config maps or secrets:

$ kubectl get -n argocd cm argocd-cm -o json | jq '.data'
{
  "repositories": "- type: git\n  url: https://github.com/argoproj/argocd-example-apps\n"
}

$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io unchanged
serviceaccount/argocd-application-controller unchanged
serviceaccount/argocd-dex-server unchanged
serviceaccount/argocd-server unchanged
role.rbac.authorization.k8s.io/argocd-application-controller unchanged
role.rbac.authorization.k8s.io/argocd-dex-server unchanged
role.rbac.authorization.k8s.io/argocd-server unchanged
clusterrole.rbac.authorization.k8s.io/argocd-application-controller unchanged
clusterrole.rbac.authorization.k8s.io/argocd-server unchanged
rolebinding.rbac.authorization.k8s.io/argocd-application-controller unchanged
rolebinding.rbac.authorization.k8s.io/argocd-dex-server unchanged
rolebinding.rbac.authorization.k8s.io/argocd-server unchanged
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/argocd-server unchanged
configmap/argocd-cm unchanged
configmap/argocd-rbac-cm unchanged
configmap/argocd-ssh-known-hosts-cm unchanged
configmap/argocd-tls-certs-cm configured
secret/argocd-secret unchanged
service/argocd-dex-server unchanged
service/argocd-metrics unchanged
service/argocd-redis unchanged
service/argocd-repo-server unchanged
service/argocd-server-metrics unchanged
service/argocd-server unchanged
deployment.apps/argocd-application-controller unchanged
deployment.apps/argocd-dex-server unchanged
deployment.apps/argocd-redis unchanged
deployment.apps/argocd-repo-server unchanged
deployment.apps/argocd-server unchanged

$ kubectl get -n argocd cm argocd-cm -o json | jq '.data'
{
  "repositories": "- type: git\n  url: https://github.com/argoproj/argocd-example-apps\n"
}

What manifests did you use?

jannfis commented 4 years ago

But actually, now as I tried to reproduce your issue, there seems to be a bug with the argocd-tls-certs-cm ConfigMap - whoops :)

jfdumont commented 4 years ago

hi jannfis

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.5.3/manifests/install.yaml

jannfis commented 4 years ago

Hi @jfdumont, these are the same manifests I used. I cannot reproduce what you reported unfortunately.

All ConfigMaps (except argocd-tks-certs-cm) are left untouched when applying manifests, and do not get reset or otherwise overwritten.

What is the version of your Kubernetes environment? Both, client and server?

jfdumont commented 4 years ago

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

But the specificity: it is a k3s cluster

jfdumont commented 4 years ago

I will do a new k3s and try to reproduce

jfdumont commented 4 years ago

On 2 different k3s clusters , same version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

1st upgrade from 1.5.0 > 1.5.4

...
configmap/argocd-cm unchanged
configmap/argocd-rbac-cm unchanged
configmap/argocd-ssh-known-hosts-cm unchanged
configmap/argocd-tls-certs-cm configured
...

2nd, the one on which the problem has already arisen, upgrade 1.5.3 > 1.5.4

...
configmap/argocd-cm configured
configmap/argocd-rbac-cm configured
configmap/argocd-ssh-known-hosts-cm unchanged
configmap/argocd-tls-certs-cm configured
...

argocd-cm , and argocd-rbac-cm are re-written

jfdumont commented 4 years ago

I found for the argocd-rbac-cm, it's related to #2931 I use rbac for user management and i apply argocd-rbac-cm.yaml following documentation, without using the last argocd-rbac-cm.yaml as source but the example, so without labels :

  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd

as we can find in last argocd-rbac-cm.yaml

In the test cluster I had not redone the rbac roles.

But for argocd-cm, I made backup before upgrade and both labels was there.

I forgot to mention ( just in case ) that I also use Argo CD Notifications.

lanmarti commented 4 years ago

I also experienced this issue upgrading from v1.5.3 to v1.5.8. Kubernetes version is v1.16.3 (both cluster and client). The following modifications were overwritten/removed:

custom known hosts added to argocd-ssh-known-hosts-cm
everything in argocd-cm, incluging: oidc.config, repository.credentials and resource.customization
custom start arguments for argocd-server (rootpath)

lanmarti commented 4 years ago

A downgrade back to v1.5.3 gave me the same behaviour. Any changes I've made in argocd-cm and argocd-ssh-known-hosts-cm have been reverted. While up/downgrading seems to overwrite the command of argocd-server, it does not seem to overwrite other changes I've made (i.e. custom volume and volumeMount, hostAliases) to the argocd-server deployment.

lanmarti commented 3 years ago

I'm still experiencing problems when upgrading. I moved from 1.5.8 to 1.7.8 today, and as in my previous comments, the contents of both argocd-cm and argocd-ssh-known-hosts are overwritten.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.13", GitCommit:"30d651da517185653e34e7ab99a792be6a3d9495", GitTreeState:"clean", BuildDate:"2020-10-15T01:06:31Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-15T01:43:56Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

upgrade statement:

$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.7.8/manifests/install.yaml

argocd-cm before upgrading

apiVersion: v1
data:
  oidc.config: # removed
  repository.credentials: |
# removed
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs
  url: # removed
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      #removed
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm

argocd-cm after upgrading

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-cm","namespace":"argocd"}}
  creationTimestamp: "2020-06-17T06:58:20Z"
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
  resourceVersion: "104272760"
  selfLink: /api/v1/namespaces/argocd/configmaps/argocd-cm
  uid: cf90dc32-5fc8-4d94-9636-0f2b7095b8b7

lanmarti commented 3 years ago

@jannfis any clue as to what may be causing this?

cruggieri commented 2 years ago

I upgraded this morning from version 2.2.0 to 2.2.4 (latest) due to the CVE of the last hours.

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

I experience this:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.2.4/manifests/install.yaml

customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io unchanged
serviceaccount/argocd-application-controller unchanged
serviceaccount/argocd-dex-server unchanged
serviceaccount/argocd-redis created
serviceaccount/argocd-server unchanged
role.rbac.authorization.k8s.io/argocd-application-controller unchanged
role.rbac.authorization.k8s.io/argocd-dex-server unchanged
role.rbac.authorization.k8s.io/argocd-server unchanged
clusterrole.rbac.authorization.k8s.io/argocd-application-controller unchanged
clusterrole.rbac.authorization.k8s.io/argocd-server unchanged
rolebinding.rbac.authorization.k8s.io/argocd-application-controller unchanged
rolebinding.rbac.authorization.k8s.io/argocd-dex-server unchanged
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server unchanged
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/argocd-server unchanged
configmap/argocd-cm unchanged
configmap/argocd-cmd-params-cm unchanged
configmap/argocd-gpg-keys-cm unchanged
configmap/argocd-rbac-cm configured
configmap/argocd-ssh-known-hosts-cm unchanged
configmap/argocd-tls-certs-cm configured
Warning: resource secrets/argocd-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
secret/argocd-secret configured
service/argocd-dex-server unchanged
service/argocd-metrics unchanged
service/argocd-redis created
service/argocd-repo-server unchanged
service/argocd-server unchanged
service/argocd-server-metrics unchanged
deployment.apps/argocd-dex-server configured
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server configured
deployment.apps/argocd-server configured
statefulset.apps/argocd-application-controller configured
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy configured
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy unchanged
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy configured
networkpolicy.networking.k8s.io/argocd-server-network-policy unchanged

Note the configmap/argocd-rbac-cm configured.

All the configuration in that config map are gone. I think this is a great issue.

jannfis commented 2 years ago

Note the configmap/argocd-rbac-cm configured.

Hm, this is weird, because in the manifests, the ConfigMap does not contain a .data section:

https://github.com/argoproj/argo-cd/blob/78d749ec88bea5a5e851fc48b9404fdf067192b6/manifests/install.yaml#L2867-L2875

arkpoah commented 2 years ago

I try to upgrade to v2.2.14 (same when trying to 2.1.16) from 2.0.1 and I get the same behavior, the diff shows my ConfigMap configurations will gone for theses ones : ConfigMap.argocd.argocd-cm ConfigMap.argocd.argocd-rbac-cm ConfigMap.argocd.argocd-ssh-known-hosts-cm

ssh-known-hosts-cm has a data section but not the two others.

deadwanderer commented 1 year ago

We've experienced the same behavior every time we've upgraded Argo (we have our self-hosted Github SSH key in known-hosts). argocd-ssh-known-hosts-cm gets overwritten (reverted to the default keys) every time we upgrade.

piotrkochan commented 1 year ago

We've experienced the same behavior every time we've upgraded Argo (we have our self-hosted Github SSH key in known-hosts). argocd-ssh-known-hosts-cm gets overwritten (reverted to the default keys) every time we upgrade.

In my case applying custom argocd-ssh-known-hosts-cm.yaml configuration has no effect and I can't see my known host in the GUI.

argoproj / argo-cd

Incomplete upgrading documentation #3537