Self managed ArgoCD out of sync after avp setup as sidecar

runderwoodcr14 commented 1 year ago

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug

I'm having a very weird problem, I'm running argocd v2.5.10 installed via the helm chart version 5.19.15, my argocd is a self managed deployment. I know that the avp support for configmap has been pushed to v2.7, however I decided that it was better to adopt the change and move to avp sidecar, so I went ahead and modified the helm chart, removed the avp plugin from the configmap and added the new sidecar configuration, pushed the changes to the repository and argo triggered a sync, all applications using avp had no problems moving to the new avp with the sidecar but argocd, I'm using avp with argocd itself to get access to our google cloud secret manager to pull secrets for connecting to the repositories, adding clusters, webhooks, etc.., that shouldn't be a problem I believe, because before the change, I was using avp as well in the same way to get credentials from gcp secret manager, now argocd is out of sync and any resources when I try to sync it doesn't update the resources, plus if I see each of the resources diff(which by the way are all the argocd resources) it shows that is OutOfSync (requires pruning). When trying to see the diff, I can only see the live manifest and it doesn't show the desired manifest, which it will render in fully removing argocd. I did a force sync and indeed argocd was removed, I deployed again argocd and all the applications just to find that the problem is still there, the only difference I can find is the avp sidecar but not sure what else can be.

To Reproduce old config using avp via configmap argo-cd helm values.yaml file

configs:
  cm:
    configManagementPlugins: |
      - name: argocd-vault-plugin-helm
        init:
          command: [sh, -c]
          args: ["helm dependency build"]
        generate:
          command: ["sh", "-c"]
          args: ["helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . --include-crds | argocd-vault-plugin generate -"]
      - name: argocd-vault-plugin-kustomize
        generate:
          command: ["sh", "-c"]
          args: ["kustomize build . | argocd-vault-plugin generate -"]

repoServer:
  env:
    - name: XDG_CONFIG_HOME
      value: /.config
    - name: AVP_TYPE
      value: gcpsecretmanager
    - name: install-ksops
      # Match Argo CD Go version
      image: viaductoss/ksops:v3.0.2
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv $GOPATH/bin/kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: argo-vault-plugin
      image: alpine:3.8
      command:
        ["/bin/sh", "-c"]
        # Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
      env:
        - name: AVP_VERSION
          value: "1.13.0"
      args:
        - >-
          wget -O argocd-vault-plugin
          https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

argocd-apps helm values.yaml file

  - name: argocd
    namespace: argocd
    destination:
      namespace: argocd
      server: https://kubernetes.default.svc
    project: argocd
    source:
      plugin:
        name: argocd-vault-plugin-helm
        env:
          - name: HELM_ARGS
            value: -f values.yaml -f ../values.override.yaml
      path: argocd/chart
      repoURL: https://gitlab.com/project/repo.git
      targetRevision: HEAD
    syncPolicy:
      syncOptions:
        - CreateNamespace=true

The new configuration

argo-cd helm values.yaml file

extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cmp-plugin
    data:
      argocd-vault-plugin-kustomize.yaml: |
        ---
        apiVersion: argoproj.io/v1alpha1
        kind: ConfigManagementPlugin
        metadata:
          name: argocd-vault-plugin-kustomize
        spec:
          allowConcurrency: true
          # Note: this command is run _before_ anything is done, therefore the logic is to check
          # if this looks like a Kustomize bundle
          discover:
            find:
              command:
                - find
                - "."
                - -name
                - kustomization.yaml
          generate:
            command:
              - sh
              - "-c"
              - "kustomize build . | argocd-vault-plugin generate -"
          lockRepo: false
      argocd-vault-plugin-helm.yaml: |
        ---
        apiVersion: argoproj.io/v1alpha1
        kind: ConfigManagementPlugin
        metadata:
          name: argocd-vault-plugin-helm
        spec:
          allowConcurrency: true
          # Note: this command is run _before_ any Helm templating is done, therefore the logic is to check
          # if this looks like a Helm chart
          discover:
            find:
              command:
                - sh
                - "-c"
                - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
          generate:
            # **IMPORTANT**: passing `${ARGOCD_ENV_helm_args}` effectively allows users to run arbitrary code in the Argo CD 
            # repo-server (or, if using a sidecar, in the plugin sidecar). Only use this when the users are completely trusted. If
            # possible, determine which Helm arguments are needed by your users and explicitly pass only those arguments.
            command:
              - sh
              - "-c"
              - |
                helm template $ARGOCD_APP_NAME --include-crds -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |
                argocd-vault-plugin generate -
          lockRepo: false
      avp.yaml: |
        apiVersion: argoproj.io/v1alpha1
        kind: ConfigManagementPlugin
        metadata:
          name: argocd-vault-plugin
        spec:
          allowConcurrency: true
          discover:
            find:
              command:
                - sh
                - "-c"
                - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
          generate:
            command:
              - argocd-vault-plugin
              - generate
              - "."
          lockRepo: false
repoServer:
  extraContainers:
    - name: argocd-vault-plugin-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.5.10
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp

        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: argocd-vault-plugin-helm.yaml
          name: cmp-plugin

        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

      # argocd-vault-plugin with Kustomize
    - name: argocd-vault-plugin-kustomize
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.5.10
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp

        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: argocd-vault-plugin-kustomize.yaml
          name: cmp-plugin

        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

      # argocd-vault-plugin with plain YAML
    - name: avp
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.5.10
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp

        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: avp.yaml
          name: cmp-plugin

        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.13.1
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
plugin sidecar
  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin

argocd-apps helm values.yaml file

  - name: argocd
    namespace: argocd
    destination:
      namespace: argocd
      server: https://kubernetes.default.svc
    project: argocd
      source:
        plugin:
          env:
            - name: AVP_TYPE
              value: gcpsecretmanager
            - name: HELM_ARGS
              value: "-f values.yaml -f ../values.override.yaml"
        path: argocd/chart
        repoURL: https://gitlab.com/project/repo.git
        targetRevision: HEAD
    syncPolicy:
      syncOptions:
        - CreateNamespace=true
  - name: applications
    namespace: argocd
    destination:
      namespace: argocd

Expected behavior selfmanaged argocd sync should not delete itself

Screenshots

Version

argocd v2.5.10+d311fad helm chart v5.19.15 Logs

Paste any relevant application logs here.

hashanmp commented 1 year ago

I faced the same issue and adding init stage to argocd-vault-plugin-helm.yaml fixed the issue for me

I'm installing argo-cd with dependencies and committed Chart.lock file on the repo as well

apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: argocd-vault-plugin-helm
spec:
  allowConcurrency: true
  discover:
    find:
      command:
        - sh
        - "-c"
        - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
  init:
    command:
      - bash
      - "-c"
      - |
        helm repo add argo https://argoproj.github.io/argo-helm
        helm dependency build 
  generate:
    command:
      - bash
      - "-c"
      - |
        helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
        argocd-vault-plugin generate -s argo-cd:argocd-vault-plugin-credentials -
  lockRepo: false

acelinkio commented 1 year ago

I believe I am facing a similar issue. Intermittently some of my argocd applications are being deleted and recreated. The only way I was able to tell this is the case is because customresource definitions inside of the applications being deleted are stuck in a deleting state.

@hashanmp could you explain how you identified your workaround as a solution? I don't quite see how it triages the problem.

hper4278 commented 1 year ago

@acelinkit in my case applications got deleted because its creating empty templates files on generate section.

helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
argocd-vault-plugin generate -s argo-cd:argocd-vault-plugin-credentials-$ARGOCD_APP_NAMESPACE -

so i had to add helm repo add argo https://argoproj.github.io/argo-helm to init and remove the chart.lock file from charts directory.

But now i have remove the find section from CMP and calling this plugin from only apps needed the plugin.

  source:
    plugin:
      name: argocd-vault-plugin-helm

    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      init:
        command:
          - bash
          - "-c"
          - |
            helm dependency update .
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s argo-cd:argocd-vault-plugin-credentials -
      lockRepo: false

argoproj / argo-cd

Self managed ArgoCD out of sync after avp setup as sidecar #12549