search

stackrox / stackrox

The StackRox Kubernetes Security Platform performs a risk analysis of the container environment, delivers visibility and runtime alerts, and provides recommendations to proactively improve security by hardening the environment.
Apache License 2.0
1.1k stars 141 forks source link

Secrets get deleted after FullSync in Argo CD #8520

Open dan-m8t opened 10 months ago

dan-m8t commented 10 months ago

Hello,

right now I am fighting with a test setup for stackrox. I am using a wrapper chart to add some more needed automation. So my Chart.yaml looks like this:

apiVersion: v2
name: stackrox-wrapper
description: Wrapper chart for stackrox
version: "1.0.0"
dependencies:
- name: stackrox-central-services
  version: "400.2.2"
  repository: "https://raw.githubusercontent.com/stackrox/helm-charts/main/opensource"

This simply pulls the Helm Chart for the central services and applies it with the value files I told Argo CD to use, this works just fine.

If I do a fullsync (which should do nothing when I didn't change anything in git obviously) the secrets needed for central, scanner and scanner-db just gets deleted:

112s        Warning   FailedMount                    pod/scanner-56d8b6b448-m54c9       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
5m46s       Warning   FailedMount                    pod/scanner-56d8b6b448-m54c9       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
92s         Warning   FailedMount                    pod/scanner-56d8b6b448-rcghv       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
5m26s       Warning   FailedMount                    pod/scanner-56d8b6b448-rcghv       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
63s         Warning   FailedMount                    pod/scanner-56d8b6b448-2wps4       MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found
4m57s       Warning   FailedMount                    pod/scanner-56d8b6b448-2wps4       MountVolume.SetUp failed for volume "scanner-tls-volume" : secret "scanner-tls" not found
2m49s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-certs-volume" : secret "central-tls" not found
4m42s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-jwt-volume" : secret "central-tls" not found
4m25s       Warning   FailedMount                    pod/central-587cc4d578-mmzsc       MountVolume.SetUp failed for volume "central-db-password" : secret "central-db-password" not found
43s         Warning   FailedMount                    pod/scanner-db-69c98c4fbf-vp7t4    MountVolume.SetUp failed for volume "scanner-db-tls-volume" : secret "scanner-db-tls" not found
43s         Warning   FailedMount                    pod/scanner-db-69c98c4fbf-vp7t4    MountVolume.SetUp failed for volume "scanner-db-password" : secret "scanner-db-password" not found

I do not see any real issue here because in the template for example for scanner-db-password are the following hooks:

    "helm.sh/hook": "pre-install,pre-upgrade"
    "helm.sh/resource-policy": keep

Afaik Argo CD respects those Helm Hooks and should NOT delete when they're annotated as mentioned.

According to the Argo CD documentation Helm Hooks are ignored when you mix them with Argo Hooks, so I migrated my Argo Hooks to Helm Hooks - first setup works as expected, but another sync with no changes still deletes the secrets and leaves the application broken :/.

Any hints here?

Thanks

dan-m8t commented 10 months ago

After some research: Argo CD does not do common helm operations like helm install or helm upgrade - but Argo CD does respect hooks (helm and argo hooks) when it is templating and applying manifests. So for Argo CD every sync is like a helm install and I guess that's why they get deleted but cannot be recreated until the deployments are ready and running. Not sure how to work around here right now.

dan-m8t commented 10 months ago

I just saw that Argo CD 2.9 supports Kustomize patches inside of a Application. As I do not want to touch the base charts I will test this out if it fits my needs and will document here.

kurlov commented 10 months ago

Thank you so much for sharing your findings!

porridge commented 5 months ago

Any updates @dan-m8t ? BTW, this sounds related to https://github.com/stackrox/stackrox/issues/2482 for which we now have a workaround in the form of https://github.com/stackrox/stackrox/pull/10300

kastl-ars commented 1 month ago

@dan-m8t Could you solve that issue?