Troubleshoot Argo Vault Plugin with Aws Secrets Manager as backend.

[pescador691]

Hi everyone!

I am trying to implement the AVP plugin using the sidecar but I am having some issues to connect it to Aws Secrets Manager. The objective is to use the AVP to retrieve the secrets from Aws to allow ArgoCD to fetch the helm charts from my private repositories.

Even thought I do not get any error, AVP cannot fetch the secrets and inject them in my argocd app. When ArgoCD syncs my application, it shows this message in the repo-server logs:

time="2023-06-22T14:14:43Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:XYZ.git,Path:cluster-k8s,TargetRevision:pedro,Helm:&ApplicationSourceHelm{ValueFiles:[],Parameters:[]HelmParameter{},ReleaseName:,Values:\"account\": \"XYZ\"\n\"clusterName\": \"XYZ\"\n\"region\": \"REGION\"\n\"repoUrl\": \"myRepo.git\"\n,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:false,IgnoreMissingValueFiles:false,SkipCrds:false,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/87f80e8af832b034b7229aa415e77b8ff72fcad2"

In my sidecar, I can see the following logs:

time="2023-06-22T14:28:38Z" level=info msg="ArgoCD ConfigManagementPlugin Server is starting" built="2023-06-20T20:51:13Z" commit=00c914a948d9e8ad99be8bd82a368fbdeba12f88 version=v2.7.6+00c914a.dirty
time="2023-06-22T14:28:38Z" level=info msg="argocd-cmp-server v2.7.6+00c914a.dirty serving on /home/argocd/cmp-server/plugins/argocd-vault-plugin-helm.sock"

In ArgoCD dashboard I got this after syncing finishes

My questions are: 1 - How can I troubleshoot this further? I do not see any error messages 2 - How do I parse the Aws credentials to AVP so it can fetch the secrets from Aws Secrets Manager?

Used the following to guide me:

• https://dev.to/luafanti/injecting-secrets-from-vault-into-helm-charts-with-argocd-49k
• https://www.cloudadmins.org/argo-vault-plugin-avp/
• https://argocd-vault-plugin.readthedocs.io/en/stable/config/

My current configuration is described below:

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
type: Opaque
stringData:
  AVP_AUTH_TYPE: "k8s"
  AVP_K8S_ROLE: "argocd-repo-server"
  AVP_TYPE: "awssecretsmanager"
  AWS_REGION: "eu-central-1"

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
  namespace: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      init:
       command:
          - bash
          - "-c"
          - |
            helm repo add bitnami https://charts.bitnami.com/bitnami
            helm dependency build      
      generate:
        command:
          - sh
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${HELM_VALUES} . |
            argocd-vault-plugin generate -s argocd:argocd-vault-plugin-credentials --verbose-sensitive-output -
      lockRepo: false

repoServer:
  envFrom: 
   - secretRef:
       name: argocd-vault-plugin-credentials
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps        
  serviceAccount:
    # service account is created by TF
    create: false
    name: "argocd-repo-server"
    automountServiceAccountToken: true
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/argocd-vault-plugin
      subPath: argocd-vault-plugin

  initContainers:
    - name: download-tools
      image: alpine:3.8
      command: [ sh, -c ]
      args:
        - >-
          wget -O argocd-vault-plugin
          https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.14.0/argocd-vault-plugin_1.14.0_linux_amd64 &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}
    - name: tmp-dir
      emptyDir: {}

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: private-helm-repository
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/sync-options: ServerSideApply=true
spec:
  destination:
    namespace: argocd
    server: {{ .Values.spec.destination.server }}
  project: default
  source:
    path: argocd-secrets
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: HELM_VALUES
          value: --namespace=argocd -f ../{{ .Values.spec.source.path }}/values.yaml
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      selfHeal: true
      prune: true

[pescador691]

Many thanks in advance!

[jvidalg]

Hi @pescador691 I am following same approach, did you ever figure it out? Thanks!