helm chart dependencies fail to download from private repo

[awalker125]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug

If using a private helm chart repo with an internally signed cert there does not seem to be away to use chart dependencies.

I can add a cert and the private repo and deploy a chart directly from the repo ok but if the chart includes a dependency it cannot download it during the helm dependency build command.

I'm following this guide https://www.openshift.com/blog/continuous-delivery-with-helm-and-argo-cd where i'd basically like to have a list of subcharts in git where I can bump the versions. This is because the deps are built and published by various third parties where I dont have the src as such.

time="2021-06-14T14:05:15Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in rd-release: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz: Get \"https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz\": x509: certificate signed by unknown authority" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-06-14T14:05:13Z" grpc.time_ms=2011.786 span.kind=server system=grpc

To Reproduce

Add a private chart repo and corresponding ca cert.

argocd cert add-tls core.harbor-${MINIKUBE_IP//./-}.nip.io --from /etc/pki/ca-trust/source/anchors/core.harbor-${MINIKUBE_IP//./-}.nip.io.crt

argocd repo add https://core.harbor-${MINIKUBE_IP//./-}.nip.io/chartrepo/example_helm --type helm --name example_helm --username admin --password ${HARBOR_ADMIN_PASSWORD}

Add a git repo with a chart like the following:

#Chart.yaml
apiVersion: v2
name: example-uber-release
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0
dependencies:
- name: example-app
  version: 1.2.10
  repository: https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm

Create an app and try to sync it.

Expected behavior

I would hope that both the login and tls cert would be used for the private repo but I guess this is maybe an edge case and argo isnt handling this. Is this supposed to work or is my approach unusual? I suppose I could get some real publicly signed certs but I'm trying a POC in minikube currently so didnt want to do that. I do wonder if it would then simply fail on authentication to the private repo anyways?

Version

(venv) [centos@tools-192-168-99-99 example-app]$ argocd version
argocd: v2.0.3+8d2b13d
  BuildDate: 2021-05-27T17:38:37Z
  GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.0.3+8d2b13d
  BuildDate: 2021-05-27T17:38:37Z
  GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

Logs

time="2021-06-14T14:05:15Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in rd-release: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz: Get \"https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz\": x509: certificate signed by unknown authority" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-06-14T14:05:13Z" grpc.time_ms=2011.786 span.kind=server system=grpc

[meerkampdvv]

i had the same issue and found a workaround by mounting the ca-store of my kubernetes nodes which includes our root ca into the pods.

For example the with the following values of the argocd helm chart :

controller:
  volumes:
    - name: ca-certificates
      hostPath:
        path: /etc/ssl/certs/ca-bundle.crt
  volumeMounts:
    - name: ca-certificates
      mountPath: /etc/ssl/certs/ca-certificates.crt

same for repoServer and Server.

we use centos7 on our hosts so your host path may vary.

[gaeljw]

I confirm the workaround described above works for https://github.com/argoproj/argo-cd/issues/13154 as well.

[todaywasawesome]

@jannfis explain issue.

[benjsc]

If you have an OCI registry for your helm chart, It appears that the reason the work around is needed is helm doesn't support adding oci repositories via "helm repo add"

Hence when argo uses: helm dependency build, helm attempts to download the oci helm chart but without a repo entry certs aren't used and it fails. If you do a helm pull, passing in the --ca-file option to helm it can successfully pull down the chart.

This actually appears a limitation of helm. Though looking at the documentation, a dependency update should work with oci registries (see https://helm.sh/docs/topics/registries/#specifying-dependencies) but the code doesn't support it with https://github.com/helm/helm/issues/12600 being the helm tracking task. Hence as of 5/Nov/23 the only work around is to override the container level CA so that openssl uses the system certs. There is no way to ignore the certs/override the certs in helm otherwise.

[ChristianCiach]

Instead of injecting the ca-bundle via ConfigMap/hostPath to /etc/ssl/certs/ca-certificates.crt, we've decided to build our own argo-cd image instead. This is our Dockerfile:

# syntax=docker/dockerfile:1.9

# image tag is automatically updated by renovate
FROM quay.io/argoproj/argocd:v2.12.2

# Changes to ca-certificates require root permissions
USER root

# Copy and install our custom root certificate
COPY  --link company-ca.crt /usr/local/share/ca-certificates/company-ca.crt
RUN <<eol
  #!/usr/bin/bash
  set -e
  chmod 644 /usr/local/share/ca-certificates/company-ca.crt
  update-ca-certificates
eol

# Change back to the argocd user. $ARGOCD_USER_ID is defined in base image:
USER $ARGOCD_USER_ID

The whole process is automated using Renovate and automerge: true, so doing this for new versions of Argo CD is mostly painless. I then use Kustomize to patch the image references from the official installation manifests:

images:
  # Use custom image with our root cert built-in:
  - name: quay.io/argoproj/argocd
    newName: our-internal-oci-repo.local/platform/ems-argo-cd