Sidecar CMP plugin cannot use private dependencies

[Blamas]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I cannot use my custom cmp sidecar plugin with private helm dependencies. It seems that the cached helm registry is not passed to the sidecar plugin. When I use the same Application without any plugin it works just fine. The plugin works when i'm not using private dependencies.

To Reproduce

With a custom sidecar cmp plugin try to use an Application with as source a git path containing an helm chart using private dependencies.

Application :

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-cmp-plugin
  namespace: argocd
spec:
  destination:
    namespace: test
    name: in-cluster
  project: default
  source:
    plugin: {}
    repoURL: https://private-repo/k8s
    path: path/application
    targetRevision: feature/debug-argo
  syncPolicy: {}

In private-repo/k8s/path/application

# Chart.yaml
apiVersion: v2
name: test
type: application
version: 0.1.0
appVersion: "0.1.0"

dependencies:
  - name: common
    repository: https://private-repo/helm/stable
    version: ">= 0.0.1"
    alias: test

---
# Values.yaml
test:
  name: test

Expected behavior

The helm chart fetch it's dependencies with the credentials from the repo server.

Version

argocd: v2.4.8+844f79e
  BuildDate: 2022-07-29T17:01:39Z
  GitCommit: 844f79eb9d8f3ab96d4ce6f8df211c6093a660ba
  GitTreeState: clean
  GoVersion: go1.18.4
  Compiler: gc
  Platform: linux/amd64

Logs

# Sidecar cmp plugin logs
time="2022-08-10T09:25:43Z" level=info msg="sh -c find . -type f -name '*.test.*' | grep '.test' && find . -type f -name 'Chart.yaml' -o -name 'values.yaml'" dir=/tmp/_cmp_server/3c6ebfae-30b3-46f7-b796-956dff66cb7c execID=ce9cc
time="2022-08-10T09:25:43Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-08-10T09:25:43Z" grpc.time_ms=3.965 span.kind=server system=grpc
time="2022-08-10T09:25:43Z" level=info msg="Generating manifests with no request-level timeout"
time="2022-08-10T09:25:43Z" level=info msg="sh -c helm dependency build" dir=/tmp/_cmp_server/ccc7bbab-0aac-431a-83f5-d08f788040af/path/application execID=acfbb
time="2022-08-10T09:25:43Z" level=error msg="`sh -c helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-ac...c1 found. (try 'helm repo update'): open /home/argocd/.cache/helm/repository/helm-manager-ac...c1-index.yaml: no such file or directory" execID=acfbb
time="2022-08-10T09:25:43Z" level=error msg="finished streaming call with code Unknown" error="error generating manifests: `sh -c helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-ac...c1 found. (try 'helm repo update'): open /home/argocd/.cache/helm/repository/helm-manager-ac...c1-index.yaml: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-08-10T09:25:43Z" grpc.time_ms=143.078 span.kind=server system=grpc

# Repo server logs
time="2022-08-10T09:25:43Z" level=info msg="manifest error cache miss: &ApplicationSource{RepoURL:https://private-repo/k8s,Path:path/application,TargetRevision:feature/debug-argo-helm,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:,Env:[]*EnvEntry{},},Chart:,}/8312f8ce54862e79c883e155a55eaa01adc220aa"
time="2022-08-10T09:25:43Z" level=error msg="finished unary call with code Unknown" error="plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-ac...c1 found. (try 'helm repo update'): open /home/argocd/.cache/helm/repository/helm-manager-ac...c1-index.yaml: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2022-08-10T09:25:42Z" grpc.time_ms=847.348 span.kind=server system=grpc

[mmalyska]

I have same problem. I have configured helm repos with credentials in argo, but cannot use them in CMP sidecar. Also similar thing with kustomize and private git repositories in resources.

[mmalyska]

@crenshaw-dev Please don't deprecate current plugin via configmap, as sidecars approach is not usable and incomplete.

[crenshaw-dev]

@mmalyska the deprecation is important, because argocd-cm plugins present major security issues. But I agree, support should not be completely removed until this issue is resolved.

[crenshaw-dev]

Related: https://github.com/argoproj/argo-cd/issues/7995

[kxs-sindrakumar]

Having the same issue, going to revert back to older version. Is there a chance a fix might be available soon @crenshaw-dev?

[yellowhat]

Hi, I am not sure if my issue is related to the above. I am able go use plugins via the "old" way by loading plugins via configMap; now I am trying to use the new way via sidecar. Even if my repository are public (my own helm chart + prometheus one) I am still getting:

time="2022-11-10T13:22:21Z" level=info msg="Generating manifests with no request-level timeout"
time="2022-11-10T13:22:21Z" level=info msg="/bin/sh -c helm dependency build" dir=/tmp/_cmp_server/94c806a6-ce48-407d-99a5-5f34df9962c5/charts/monitoring execID=91f27
time="2022-11-10T13:22:22Z" level=error msg="`/bin/sh -c helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-a3d71d79da3a7e97fc44127f373aa765c1b71499cb48cdf46a0c10608bd437bc found. (try 'helm repo update'): open /.cache/helm/repository/helm-manager-a3d71d79da3a7e97fc44127f373aa765c1b71499cb48cdf46a0c10608bd437bc-index.yaml: no such file or directory" execID=91f27
time="2022-11-10T13:22:22Z" level=error msg="finished streaming call with code Unknown" error="error generating manifests: `/bin/sh -c helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-a3d71d79da3a7e97fc44127f373aa765c1b71499cb48cdf46a0c10608bd437bc found. (try 'helm repo update'): open /.cache/helm/repository/helm-manager-a3d71d79da3a7e97fc44127f373aa765c1b71499cb48cdf46a0c10608bd437bc-index.yaml: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-11-10T13:22:21Z" grpc.time_ms=804.455 span.kind=server system=grpc

Should the cache be mounted across the 2 containers?

Thanks

[crenshaw-dev]

@kxs-sindrakumar we won't be able to revert, because the solution I merged was insecure. We can't allow all CMPs full access to repo creds. There needs to be some kind of system for an admin to permit CMPs to access secrets for certain repos.

@yellowhat this looks like a different issue, can you open a new one? I don't think the Helm cache needs to be shared, but I could be wrong about that. Maybe just need to mount an emptyDir for the Helm cache, similar to how it's done on the main repo-server container. But I'd avoid sharing resources unless absolutely necessary.

[yellowhat]

Are you referring to this location?

[mmalyska]

@yellowhat You need to fetch dependency yourself in plugin helm dependency update or when doing the template you need to use option --dependency-update ex. helm template . --dependency-update to make help auto fetch dependencies in plugin(because there is no share of helm cache in plugin sidecar).

[yellowhat]

Yes the init.command is helm dependency update. I will try to mount the helm-working-dir volume.

[yellowhat]

I can confirm that was due to a missing folder. I am assuming that it is looking for the env var HELM_CACHE_HOME, HELM_CONFIG_HOME, HELM_DATA_HOME. The following works for me:

repoServer:
  replicas: 1
  extraContainers:
    - name: cmp
      command: ["/var/run/argocd/argocd-cmp-server"]
      # Use argocd as it contains helm and kustomize
      image: quay.io/argoproj/argocd:latest
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      env:
        - name: HELM_CACHE_HOME
          value: /helm-working-dir
        - name: HELM_CONFIG_HOME
          value: /helm-working-dir
        - name: HELM_DATA_HOME
          value: /helm-working-dir
      volumeMounts:
        - name: var-files
          mountPath: /var/run/argocd
        - name: plugins
          mountPath: /home/argocd/cmp-server/plugins
        - name: cmp-plugin
          mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: plugin.yaml
        - name: cmp-tmp
          mountPath: /tmp
        - name: helm-temp-dir
          mountPath: /helm-working-dir
  volumes:
    - name: cmp-plugin
      configMap:
        name: cmp-plugin
    - name: cmp-tmp
      emptyDir: {}
    # avoid to share the volume helm-working-dir
    - name: helm-temp-dir
      emptyDir: {}

I think the documentation should include an example for helm/kustomize usecases.

[crenshaw-dev]

@yellowhat I'd be happy to review a docs PR if you have a chance to write it up!

Have you tried mounting a different Helm temp dir for the sidecar vs. the one used by the main repo-server container?

The point of the sidecars is to be as stateless/ephemeral and share as little information as possible with the main container.

[yellowhat]

As you can see in my example above, I am mounting helm-temp-dir not helm-working-dir volume (the one mounted on the repo-server container). My usecase seems to work with either volumes. I am not sure if sharing helm-working-dir would be required for a private repository.

[kxs-sindrakumar]

@crenshaw-dev how are we supposed to use this side car for kustomize if we have a private repo to another private repo. When CMP runs kustomize build on the first repo, it doesn't seem to generate the manifest. The secrets for both repos are the same

[hamza-tumturk]

@crenshaw-dev how are we supposed to use this side car for kustomize if we have a private repo to another private repo. When CMP runs kustomize build on the first repo, it doesn't seem to generate the manifest. The secrets for both repos are the same

Workaround for the meantime: I managed to get it working by mounting the ssh credentials for pulling the private repo in the argocd-repo-server. Created ssh configmaps and secrets and then mounted them all. Here is an example patch:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: argocd-repo-server
  spec:
    template:
      spec:
        volumes:
        - name: ssh-config
          configMap:
            name: argocd-ssh-config
        - name: ssh-key
          secret:
            secretName: argocd-ssh
        - name: ssh-known-hosts
          configMap:
            name: argocd-ssh-known-hosts-cm
        containers:
        - name: argocd-repo-server
          volumeMounts:
          - mountPath: /home/argocd/.ssh/config
            name: ssh-config
            subPath: ssh-config
          - mountPath: /home/argocd/.ssh/id_rsa
            name: ssh-key
            subPath: argocd-ssh
          - mountPath: /home/argocd/.ssh/known_hosts
            name: ssh-known-hosts
            subPath: ssh_known_hosts

Then create a secret ssh-key with the pem and set ssh-config configmap like

Host repository.org
  IdentityFile ~/.ssh/id_rsa

The configmap ssh-known-hosts already exists as part of the ArgoCD deployment.

[jemag]

If deprecation of CMP through argocd-cm is to be done by 2.7, I feel like this issue should also target milestone v2.7.

Or, at the very least, documentation should be provided of a satisfactory workaround.

[crenshaw-dev]

@jemag this limitation is not unique to sidecar CMPs. Starting with (iirc) 2.3, no plugins are given access to Argo CD's repo credentials.

[jemag]

hmm that is weird. I currently use a kustomized-helm plugin through the configmap with private repositories and haven't faced any issues. Haven't configured any particular workaround either, just using the classic:

  configManagementPlugins: |
    - name: kustomized-helm
      init:
        command: ["/bin/sh", "-c"]
        args: ["helm dependency build || true"]
      generate:
        command: ["/bin/sh", "-c"]
        args: ["helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds -f $ARGOCD_ENV_VALUES_PATH > all.yaml && kustomize build $ARGOCD_ENV_OVERLAY_PATH"]

[crenshaw-dev]

Hm... there's a pretty good chance I'm just straight-up wrong on this one. I need to dive in and make sure what the current state is. Won't be able to do that for a little while though. :-(

[jemag]

just as a clarification on my comment @crenshaw-dev , it worked with 2.5.1, however it indeed does not seem to work anymore if I upgrade past 2.5.2 (currenty experimenting with 2.6.0), see https://github.com/argoproj/argo-cd/issues/11747

Not sure if there is a breaking change notice that I missed, but seems weird that this would change within 2.5.x

[msobkowiak-olx]

I must say we ran into this problem when upgrading from 2.7.3 to 2.8.2, this only occurs if the helm chart itself is loaded from an external private repository (locally cached helm charts worked as usual, also plugins like helm-secrets worked too):

no cached repository for helm-manager-6d77cc5d2b67cb858de97b5e4ca5537b886f133c28a9a2824bc8122543bd28d4 found. 
(try 'helm repo update'): open /helm-working-dir/repository/helm-manager-6d77cc5d2b67cb858de97b5e4ca5537b886f133c28a9a2824bc8122543bd28d4-index.yaml: no such file or directory"

I am trying to check our CMP custom image for any misconfigurations as well as initcontainer for repoServer (we hit this on RepoServer it seems).

EDIT:

I must say our initContainer setup for RepoServer does not follow the official docs anymore as we are setting HELM_PLUGINS env ourselves and nothing else (no cache dir or other ones listed in the documentation for 2.8.2):

  env:
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/

And then installing plugins like so:

      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

I will give it a go to refactor the initContainer to be in tune with the official guide and set all the vars described there: https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#using-initcontainers

[msobkowiak-olx]

Well, after some testing we found out that the issue occurs for us (again, when migrating from 2.7.3 to 2.8.x) when you have dependency in requirements.yaml

dependencies:
  - name: xxx
    version: x.y.z
    repository: https://example.com/helm

What works is adding this section to chart.yaml:

apiVersion: v1
name: myapp
version: 1.0.0
dependencies:
  - name: xxx
    version: x.y.z
    repository: https://example.com/helm

[Cyben]

Hey, is there a solution for this?