Secrets get deleted after FullSync in Argo CD

[dan-m8t]

Hello,

right now I am fighting with a test setup for stackrox. I am using a wrapper chart to add some more needed automation. So my Chart.yaml looks like this:

apiVersion: v2
name: stackrox-wrapper
description: Wrapper chart for stackrox
version: "1.0.0"
dependencies:
- name: stackrox-central-services
  version: "400.2.2"
  repository: "https://raw.githubusercontent.com/stackrox/helm-charts/main/opensource"

This simply pulls the Helm Chart for the central services and applies it with the value files I told Argo CD to use, this works just fine.

If I do a fullsync (which should do nothing when I didn't change anything in git obviously) the secrets needed for central, scanner and scanner-db just gets deleted:

112s        Warning   FailedMount                    pod/scanner-56d8b6b448-m54c9       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
5m46s       Warning   FailedMount                    pod/scanner-56d8b6b448-m54c9       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
92s         Warning   FailedMount                    pod/scanner-56d8b6b448-rcghv       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
5m26s       Warning   FailedMount                    pod/scanner-56d8b6b448-rcghv       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
63s         Warning   FailedMount                    pod/scanner-56d8b6b448-2wps4       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
4m57s       Warning   FailedMount                    pod/scanner-56d8b6b448-2wps4       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
2m49s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-certs-volume" : secret "central-tls" not found
4m42s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-jwt-volume" : secret "central-tls" not found
4m25s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-db-password" : secret "central-db-password" not found
43s         Warning   FailedMount                    pod/scanner-db-69c98c4fbf-vp7t4    MountVolume.SetUp failed for volume "scanner-db-tls-volume" : secret "scanner-db-tls" not found
43s         Warning   FailedMount                    pod/scanner-db-69c98c4fbf-vp7t4    MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found

I do not see any real issue here because in the template for example for scanner-db-password are the following hooks:

    "helm.sh/hook": "pre-install,pre-upgrade"
    "helm.sh/resource-policy": keep

Afaik Argo CD respects those Helm Hooks and should NOT delete when they're annotated as mentioned.

According to the Argo CD documentation Helm Hooks are ignored when you mix them with Argo Hooks, so I migrated my Argo Hooks to Helm Hooks - first setup works as expected, but another sync with no changes still deletes the secrets and leaves the application broken :/.

Any hints here?

Thanks

[dan-m8t]

After some research: Argo CD does not do common helm operations like helm install or helm upgrade - but Argo CD does respect hooks (helm and argo hooks) when it is templating and applying manifests. So for Argo CD every sync is like a helm install and I guess that's why they get deleted but cannot be recreated until the deployments are ready and running. Not sure how to work around here right now.

[dan-m8t]

I just saw that Argo CD 2.9 supports Kustomize patches inside of a Application. As I do not want to touch the base charts I will test this out if it fits my needs and will document here.

[kurlov]

Thank you so much for sharing your findings!

[porridge]

Any updates @dan-m8t ? BTW, this sounds related to https://github.com/stackrox/stackrox/issues/2482 for which we now have a workaround in the form of https://github.com/stackrox/stackrox/pull/10300

[kastl-ars]

@dan-m8t Could you solve that issue?