ArgoCD vault Plugin: line 5: syntax error near unexpected token `]' and Hashicorp Vault cannot retrieve cached token: stat /.avp/config.json:

Describe the bug After a recent upgrade of ArgoCD operator and argoCD vault plugin on Openshift Cluster 4.16.15, we are seeing this error. Apps on the argocd are going into unknown state throwing an error as below

I'm not sure if the issue is with AVP or argoCD. I was asked to open an issue here as well.

Original issue: https://github.com/argoproj/argo-cd/issues/20673

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `bash -c "helm template \"$ARGOCD_APP_NAME\" -n \"$ARGOCD_APP_NAMESPACE\" -f $(echo \"$ARGOCD_ENV_HELM_VALUES\") . | argocd-vault-plugin generate -"` failed exit status 2: /usr/local/bin/argocd-vault-plugin: line 5: syntax error near unexpected token `]' /usr/local/bin/argocd-vault-plugin: line 5: ` } ]'

Here is the cmp-plugin.yaml Config map


kind: ConfigMap
apiVersion: v1
metadata:
  name: cmp-plugin
  namespace: openshift-gitops
  uid: 218a3b09-3a33-4438-9577-b520f1bd8266
  resourceVersion: '516875537'
  creationTimestamp: '2023-09-29T20:51:05Z'
  labels:
    app.kubernetes.io/instance: argocd-gitops-local-cluster
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"avp.yaml":"apiVersion: argoproj.io/v1alpha1\nkind: ConfigManagementPlugin\nmetadata:\n  name: argocd-vault-plugin\nspec:\n  allowConcurrency: true\n  generate:\n    command: [\"bash\", \"-c\"]\n    args: ['helm template \"$ARGOCD_APP_NAME\" -n \"$ARGOCD_APP_NAMESPACE\" -f $(echo \"$ARGOCD_ENV_HELM_VALUES\") . | argocd-vault-plugin generate -']\n  lockRepo: false\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"argocd-gitops-local-cluster"},"name":"cmp-plugin","namespace":"openshift-gitops"}}
  managedFields:
    - manager: argocd-controller
      operation: Update
      apiVersion: v1
      time: '2023-10-02T15:28:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data': {}
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2024-10-29T22:40:35Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          'f:avp.yaml': {}
data:
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      generate:
        command: ["bash", "-c"]
        args: ['helm template "$ARGOCD_APP_NAME" -n "$ARGOCD_APP_NAMESPACE" -f $(echo "$ARGOCD_ENV_HELM_VALUES") . | argocd-vault-plugin generate -']
      lockRepo: false

To reproduce

There are few apps in argocd which are failing with the above error and the helm charts havent changed

Expected behavior The apps on the argocd UI should sync normally and shouldn't go into unknown state throwing the above error

Screenshots/Verbose output

oc rsh -c avp openshift-gitops-repo-server-58fcd5cdc4-s8dmw
sh-4.4$ argocd-vault-plugin version
argocd-vault-plugin v1.18.1 (fc452cdd8d4727b412ce3de61ee0416efd75050d) BuildDate: 2024-06-07T03:17:37Z

$ argocd-vault-plugin generate vault.yaml --verbose-sensitive-output
2024/11/19 23:14:05 reading configuration from environment, overriding any previous settings
2024/11/19 23:14:05 AVP configured with the following settings:

2024/11/19 23:14:05 avp_kv_version: 2

2024/11/19 23:14:05 Hashicorp Vault cannot retrieve cached token: stat /.avp/config.json: no such file or directory. Generating a new one

If you've tried running argocd-vault-plugin generate with --verbose-sensitive-output to help debug, please include that output here after redacting any secrets.

Additional context

Here is argocd and avp install as sidecar container

---
apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  finalizers:
    - argoproj.io/finalizer
  name: openshift-gitops
spec:
  controller:
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: 250m
        memory: 1Gi
  server:
    ingress:
      enabled: false
    route:
      enabled: true
  rbac:
    defaultPolicy: ''
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
      g, OpenShift Admins, role:admin
    scopes: '[groups]'
  repo:
    serviceaccount: default
    mountsatoken: true
    volumes:
      - configMap:
          name: cmp-plugin
        name: cmp-plugin
      - name: custom-tools
        emptyDir: {}
    initContainers:
      - args:
        - -c
        - curl -k -L -artuser:${ARTIFACTORY_API_KEY} https://artifactory.domain:443/artifactory/devops-local/downloads/argocd-vault-plugin/argocd-vault-plugin_${VAULT_PLUGIN_VERSION}_linux_amd64
          -o /custom-tools/argocd-vault-plugin && chmod +x /custom-tools/argocd-vault-plugin
        command:
        - /bin/bash
        env:
        - name: ARTIFACTORY_API_KEY
          valueFrom:
            secretKeyRef:
              key: ARTIFACTORY_API_KEY
              name: artifactory
        - name: VAULT_PLUGIN_VERSION
          value: 1.18.1
        image: registry.domain/ubi8
        name: download-tools
        resources: {}
        volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    sidecarContainers:
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        envFrom:
          - secretRef:
              name: vault-configuration
        image: registry.domain/openshift-gitops-1/argocd-rhel8:v1.14.1
        securityContext:
          runAsNonRoot: true
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp
          - name: tls-certs
            mountPath: /app/config/tls/
          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp.yaml
            name: cmp-plugin
          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin
  sso:
    dex:
      openShiftOAuth: true
      resources:
        limits:
          cpu: 500m
          memory: 256Mi
        requests:
          cpu: 250m
          memory: 128Mi
    provider: dex

argoproj-labs / argocd-vault-plugin

ArgoCD vault Plugin: line 5: syntax error near unexpected token `]' and Hashicorp Vault cannot retrieve cached token: stat /.avp/config.json: #675