argocd-vault-plugin with values.yaml in other git repo

[en1cc]

Describe the bug When I deploy an application with helm, I use the following template:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-dns
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: workshop
  sources:
  - repoURL: 'https://charts.bitnami.com/bitnami'
    chart: external-dns
    targetRevision: 6.*
    helm:
      valueFiles:
      - $values/argocd/helm-values/external-dns/values.yaml
  - repoURL: 'https://gitlab.com/some/project.git'
    targetRevision: HEAD
    ref: values
  destination:
    name: in-cluster
    namespace: external-dns
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true

I use the upstream Chart and add a values file from my git repo. How can I achieve this with argocd-vault-plugin running as sidecar? I dont want to write the whole values file as extra env args to the application deployment

Expected behavior I expect the plugin to work like this:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-dns
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://charts.bitnami.com/bitnami
      chart: external-dns
      targetRevision: 6.31.4
      plugin:
        env:
          - name: HELM_ARGS
            value: -f $values/helm/external-dns/values.yaml
    - repoURL: git@gitlab.com/some/project.git
      targetRevision: main
      ref: values
  destination:
    name: in-cluster
    namespace: external-dns

Does anyone deploy like this? I dont want to use an umbrella chart, because then I have to hardcode the repositorys at the init stage or checking the whole upstream helm chart in my git repo.

Best, Patrick

[en1cc]

No one who has the same issues? How do you use this plugin with values? Any recommendations?

[cdallaire81]

I found a workaround where I use a chart with a subchart. So my argocd application points to a git repo where my values.yaml file is but I've also got a Chart.yaml with a dependancy chart. See https://levelup.gitconnected.com/helm-dependencies-1907facbe410

Then as per this issue I also had to modify the plugin's behaviour to include the command : helm dependency build. See https://github.com/argoproj/argo-cd/issues/13009

[en1cc]

I found a workaround where I use a chart with a subchart. So my argocd application points to a git repo where my values.yaml file is but I've also got a Chart.yaml with a dependancy chart. See https://levelup.gitconnected.com/helm-dependencies-1907facbe410

Then as per this issue I also had to modify the plugin's behaviour to include the command : helm dependency build. See argoproj/argo-cd#13009

Can you share your Application.yaml for ArgoCD? I want to have a look how you definded the application

[cdallaire81]

source:
    repoURL: https://bitbucket.org/bbbbbb/my-cicd-tests.git
    path: avp-tests 
    plugin:
      name: avp-helm

In the folder avp-tests I have Chart.yaml and values.yaml

My avp plugin assumes my values file is called values.yaml otherwise I could have passed it to the plugin as an argument

[cdallaire81]

And here's my plugin

avp-helm: 
        # Note: this command is run _before_ any Helm templating is done, therefore the logic is to check
        # if this looks like a Helm chart
        discover:
          find:
            command:
              - sh
              - "-c"
              - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
        init:
          command: 
            - sh
            - "-c"
            - |
              helm dependency build
        generate:
          # **IMPORTANT**: passing `${ARGOCD_ENV_HELM_ARGS}` effectively allows users to run arbitrary code in the Argo CD 
          # repo-server (or, if using a sidecar, in the plugin sidecar). Only use this when the users are completely trusted. If
          # possible, determine which Helm arguments are needed by your users and explicitly pass only those arguments.
          command:
            - sh
            - "-c"
            - |
              argocd-vault-plugin generate values.yaml > generated-values.yaml && helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f generated-values.yaml . --include-crds 
        lockRepo: false

[rothbauer]

I need this as well, I have values.yaml and multiple override files that needs to be merged and secrets from Vault injected to them... it's basically the same as @en1cc's example only with that difference that I have multiple valueFiles.

[w0aw]

same issue here. don't know how to use and even which plugin to use.

my application file -

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
   name: cert-manager
   namespace: gitops
spec:
   project: default
   destination:
      server: https://kubernetes.default.svc
      namespace: security
   sources:
      - repoURL: "https://charts.jetstack.io"
        targetRevision: 1.14.5
        chart: cert-manager
        helm:
           valueFiles:
              - $rootDir/initial resources/cert-manager/values.yaml
      - repoURL: "git@github.com:my-org/helm-values.git"
        plugin:
           name: argocd-vault-plugin
        targetRevision: main
        ref: rootDir
   syncPolicy:
      automated:
         prune: true
         selfHeal: true
      syncOptions:
         - CreateNamespace=true

and here i have my values.yaml file for cert manager.

installCRDs: true

webhook:
   hostNetwork: true
   securePort: 19097

extraEnv:
   - name: TEST_ARGOCD_VAULT_1
     value: <path:secrets-kv2/data/argocd#test>
   - name: TEST_ARGOCD_VAULT_2
     value: <path:secrets-kv2/argocd#test>

and here i have some logs from my vault pod

vault@vault-server-0:/$ curl --request POST   -H "Content-Type: application/json"   -d '{"jwt": "'$JWT'", "role": "argocd"}'   http://127.0.0.1:8200/v1/auth/kubernetes/login

{
   "request_id": "7ac62316-2a60-3555-5f42-85df7c554048",
   "lease_id": "",
   "renewable": false,
   "lease_duration": 0,
   "data": null,
   "wrap_info": null,
   "warnings": null,
   "auth": {
      "client_token": "hvs.CAESILNzaHMbitgN1KaIsiSyjdTlHbfp8kFO39t5EhqbZsFTGh4KHGh2cy5najJValUxOEttOWtmUlpjTVI1YTlLdkg",
      "accessor": "CW21WDy4kBnCXxww7DNyZqE0",
      "policies": ["argocd-kv-role", "default"],
      "token_policies": ["argocd-kv-role", "default"],
      "metadata": {
         "role": "argocd",
         "service_account_name": "argocd-repo-server",
         "service_account_namespace": "gitops",
         "service_account_secret_name": "",
         "service_account_uid": "a8942c51-6bcd-4c06-8fb7-ae0dd219c457"
      },
      "lease_duration": 86400,
      "renewable": true,
      "entity_id": "0bfa6385-9191-c895-bb36-529c4fdadff4",
      "token_type": "service",
      "orphan": true,
      "mfa_requirement": null,
      "num_uses": 0
   },
   "mount_type": ""
}

vault@vault-server-0:/$ vault kv get secrets-kv2/argocd
===== Secret Path =====
secrets-kv2/data/argocd

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-15T07:01:36.073850515Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key     Value
---     -----
test    working

and here i have config for installing official argocd helm chart

configs:
   params:
      server.insecure: true
   cmp:
      create: true
      plugins:
         argocd-vault-plugin:
            allowConcurrency: true
            discover:
               find:
                  command:
                     - sh
                     - "-c"
                     - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
            generate:
               command:
                  - argocd-vault-plugin
                  - generate
                  - "."
                  - "-s"
                  - "argocd-vault-plugin-configuration"
            lockRepo: false

applicationSet:
   enabled: false

dex:
   enabled: false

controller:
   replicas: 2

repoServer:
   rbac:
      - verbs:
           - get
           - list
           - watch
        apiGroups:
           - ""
        resources:
           - secrets
           - configmaps
   volumes:
      - name: argocd-cmp-cm
        configMap:
           name: argocd-cmp-cm
      - name: custom-tools
        emptyDir: {}

   initContainers:
      - name: download-tools
        image: registry.access.redhat.com/ubi8
        env:
           - name: AVP_VERSION
             value: 1.16.1
        command: [sh, -c]
        args:
           - >-
              curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
              chmod +x argocd-vault-plugin &&
              mv argocd-vault-plugin /custom-tools/
        volumeMounts:
           - mountPath: /custom-tools
             name: custom-tools
   extraContainers:
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        image: registry.access.redhat.com/ubi8
        securityContext:
           runAsNonRoot: true
           runAsUser: 999
        volumeMounts:
           - mountPath: /var/run/argocd
             name: var-files
           - mountPath: /home/argocd/cmp-server/plugins
             name: plugins
           - mountPath: /tmp
             name: tmp

           # Register plugins into sidecar
           - mountPath: /home/argocd/cmp-server/config/plugin.yaml
             subPath: argocd-vault-plugin.yaml
             name: argocd-cmp-cm

           # Important: Mount tools into $PATH
           - name: custom-tools
             subPath: argocd-vault-plugin
             mountPath: /usr/local/bin/argocd-vault-plugin
notifications:
   enabled: true

redis:
   enabled: true

# vault config
extraObjects:
   - apiVersion: v1
     kind: Secret
     type: Opaque
     metadata:
        name: argocd-vault-plugin-configuration
     data:
        VAULT_ADDR: aHR0cDovL3ZhdWx0LXNlcnZlci1oZWFkbGVzcy5zZWN1cml0eQ==
        AVP_AUTH_TYPE: azhz
        AVP_K8S_ROLE: YXJnb2Nk
        AVP_TYPE: dmF1bHQ=

here is end result of env vars on cert manager TEST_ARGOCD_VAULT_1 : <path:secrets-kv2/data/argocd#test> TEST_ARGOCD_VAULT_2 : <path:secrets-kv2/argocd#test>

there is no error in logs or repo server and plugin sidecar.

Please let me know if i am doing somethiong wrong

config 2

after trying something new with application file -

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
   name: cert-manager
   namespace: gitops
spec:
   project: default
   destination:
      server: https://kubernetes.default.svc
      namespace: security
   sources:
      - repoURL: "https://charts.jetstack.io"
        targetRevision: 1.14.5
        chart: cert-manager
        helm:
           valueFiles:
              - $rootDir/initial resources/cert-manager/values.yaml
        plugin:
           name: argocd-vault-plugin-helm
           env:
              - name: HELM_VALUES
                value: |
                   extraEnv:
                       - name: TEST_ARGOCD_VAULT_3
                         value: <path:secrets-kv2/data/argocd#test>
      - repoURL: "git@github.com:my-repo/helm-values.git"
        targetRevision: main
        ref: rootDir
   syncPolicy:
      automated:
         prune: true
         selfHeal: true
      syncOptions:
         - CreateNamespace=true

finally i see some error in UI -

Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = error getting app source type: multiple application sources defined: Helm,Plugin

i guess this is still not supported

[Stefanqn]

tbh I'm not sure, if that's an AVP issue: it seems that all sources containing ref: aren't handed to CMPs and setting the plugin field on an helm source throws an "multiple application sources defined: Helm,Plugin" error. Both looks like restrictions by ArgoCDs Repo server.