After upgrade to 2.11.3 couldn't find cmp-server plugin

[YuriiBudnyi]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug After upgrading ArgoCD from version 2.10.2 to 2.11.3, we encountered an issue with one of our applications. When attempting to change the target revision using the ArgoCD UI, the following error occurs:

level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for app_name is invalid: InvalidSpecError: Unable to generate manifests in deploy: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. couldn't find cmp-server plugin with name \"tanka\" supporting the given repository" grpc.code=InvalidArgument grpc.method=Update grpc.service=application.ApplicationService grpc.start_time="2024-06-24T11:55:45Z" grpc.time_ms=1435.673 span.kind=server system=grpc

All others apps works fine and I can change target revision branch name via UI.

Additional Context: Attempted to delete and reapply the application. Restarted Redis for caching. Restarted repo-server pods which have the sidecar tanka plugin.

Manifest of app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: 'dev:argoproj.io/Application:argocd/dev-web'
    janitor/ttl: 8h
  name: dev
  namespace: argocd
spec:
  destination:
    namespace: dev
    server: 'https://kubernetes.default.svc'
  project: dev
  source:
    path: deploy
    plugin:
      env:
        - name: TK_ENV
          value: environments/dev
        - name: TAG_SRC
          value: ':(exclude)deploy'
      name: tanka
    repoURL: 'https://github.com/...'
    targetRevision: environment/dev
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=false

To Reproduce Not sure how to reproduce it. Happens with only one app.

1. Upgrade ArgoCD to version 2.11.3 from 2.10.2.
2. Navigate to the ArgoCD UI.
3. Select an application.
4. Attempt to change the target revision using the UI. Observe the error mentioned above.

Expected behavior

Screenshots

Version 2.11.3

Logs

level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for app_name is invalid: InvalidSpecError: Unable to generate manifests in deploy: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. couldn't find cmp-server plugin with name \"tanka\" supporting the given repository" grpc.code=InvalidArgument grpc.method=Update grpc.service=application.ApplicationService grpc.start_time="2024-06-24T11:55:45Z" grpc.time_ms=1435.673 span.kind=server system=grpc

[christianh814]

Are you still having this issue? The log indicates that it doesn't know about the cmp "tanka", did you verify that it's in the right ns? (I know this was an upgrade but just trying to be thorough)

[crenshaw-dev]

Can you also post your plugin config?

[YuriiBudnyi]

Hi @christianh814. Yes in the same namespace, in argocd ns, like other applications. It is an issue with only one app.

apiVersion: v1
data:
  tanka.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: tanka
    spec:
      generate:
        command:
        - /usr/local/bin/tanka.sh
kind: ConfigMap
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: ''
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helmraiser
    app.kubernetes.io/name: argocd-cmp-cm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.11.3
    helm.sh/chart: argo-cd-7.1.4
  name: argocd-cmp-cm
  namespace: argocd

tanka.sh (removed some field)

    #!/usr/bin/env bash
    set -e
    jb install
    tk show "path_main.jsonnet" --dangerous-allow-redirect -V tag=image_tag