Provide a easy way to add a custom CA

[klausenbusk]

Summary

Argo supports adding a custom TLS certificate per repository, but it isn't very scalable if you ex: want to MITM all traffic.

Motivation

In our case we want to MITM all traffic with a proxy and we don't necessarily know all server names up front. Adding a TLS certificate per server name is cumbersome and the only alternative seems to BYOI (add the certificate and run update-ca-certificates).

Proposal

We could perhaps use the existing infrastructure and support a "default certificate" in: https://github.com/argoproj/argo-cd/blob/b37eee1054e42c873699460dd5e2447c2f9fe5a6/util/cert/cert.go#L321-L334

[jgallucci32]

This can be achieved by creating a Kubernetes secret and adding it as volume mounts to the argo-cd pods. Since ArgoCD is written in Go it will automatically look for the custom CA certs placed in /etc/ssl/certs/. Here are the steps:

1. Create a Kubernetes secret called custom-ca-certificates using your custom CA bundle:

   kubectl create secret generic custom-ca-certificates --from-file=custom-ca-certificates.crt=/path/to/custom/bundle.crt

2. Add volume and volumeMount parameters to the Helm values.yaml answer file:

   controller:
   volumeMounts:
     - name: custom-ca-certificates
       mountPath: /etc/ssl/certs/custom-ca-certificates.crt
       subPath: custom-ca-certificates.crt
   volumes:
     - name: custom-ca-certificates
       secret:
         defaultMode: 420
         secretName: custom-ca-certificates
   repoServer:
   volumeMounts:
     - name: custom-ca-certificates
       mountPath: /etc/ssl/certs/custom-ca-certificates.crt
       subPath: custom-ca-certificates.crt
   volumes:
     - name: custom-ca-certificates
       secret:
         defaultMode: 420
         secretName: custom-ca-certificates
   server:
   volumeMounts:
     - name: custom-ca-certificates
       mountPath: /etc/ssl/certs/custom-ca-certificates.crt
       subPath: custom-ca-certificates.crt
   volumes:
     - name: custom-ca-certificates
       secret:
         defaultMode: 420
         secretName: custom-ca-certificates

3. Upgrade Helm chart and it now can use custom certificates from your SSL proxy or other sites with custom signed certificates.

[Morriz]

The operator has no such fields. Any insights?

[WesselAtWork]

I had multiple certs (root and intermediaries) so I mounted the certs on /etc/ssl/certs. Sadly that overrides EVERYTHING on that. Everything was fine till I needed to do helm charts. Our helm charts uses some upstream charts and while connecting to our main repo worked fine, connecting to the upstream failed. Fixed it with:

initContainers:
  name: generate-certs
  image: docker.io/alpine/curl:latest
  command:
    - "/usr/sbin/update-ca-certificates"
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    runAsNonRoot: false
    allowPrivilegeEscalation: true
    readOnlyRootFilesystem: false
    capabilities:
      add:
      - ALL
  volumeMounts:
   - name: ssl-certs
      mountPath: /etc/ssl/certs
   - name: ssl-local
      mountPath: /usr/local/share/ca-certificates
  volumes:
   - name: ssl-certs
     emptyDir: {}
   - name: ssl-local
     emptyDir: {}

I put this init job on every application that required the certs:

• Controller
• ApplicationSet Controller
• Server
• Repo Server
• Notification Server (unsure if this one is actually required)

I don't think you need the insanely privileged securityContext. I was struggling getting it to work because I accidentally mounted /usr/share/ca-certificates which overode the /mozilla which was were all the default certs were and update-ca-certificates --fresh wasn't working the way I thought it was. If you can do it with less privileges (or specific privileges) I'd love to see them.

[lknite]

I've been using the work around to use a mount to overwrite /etc/ssl/certs but this is not a full solution. Overwriting /etc/ssl/certs with my onprem certs means that some certs normally verified with the default /etc/ssl/certs were are no longer verified, such as dev.azure.com in my situation. I can add them manually but eventually they expire.

In the past I have copied the ca-certificates bundle which comes with linux by default, appended my onprem certs, then added this to a configmap or secret, but it turned out the file was too big so this work-around does not always work.

We need a way to specify onprem certs that will then be added using whatever method the source argocd image uses such as 'update-ca-trust' or 'update-ca-certificates' after adding to an import directory.

[lknite]

I wrote up a way to implement this at another site, maybe we could use the same technique: https://github.com/bitnami/charts/issues/19744

global:
  # specify by providing the name of a secret and use all data as .crt files
  ca-bundle-secret: ca-bundle
  # or specify directly
  ca-bundle: |
    -----BEGIN CERTIFICATE-----
    MIIFrDCCBJSgAwIBAgIQCfluwpVVXyR0nq8eXc7UnTANBgkqhkiG9w0BAQwFADBh
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIFrDCCBJSgAwIBAgIQCfluwpVVXyR0nq8eXc7UnTANBgkqhkiG9w0BAQwFADBh
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
    -----END CERTIFICATE-----

[lknite]

Summary

Here's a pvc-based workaround inspired by @WesselAtWork (needed to get around size limitations of a configmap / secret, otherwise we could just mount one of those):

1. mount one or more certs from a ca-bundle secret to the /tmp/source folder
2. loop through certs and split into multiple cert files as needed, save to /usr/local/share/ca-certificates
3. run update-ca-certificates to import the additional certs
4. copy the /etc/ssl/certs folder contents to /tmp/target (which is the pvc)
5. mount the pvc to /etc/ssl/certs in the pod

Requirements

• RWX required OR:
  • One pod mounts pvc as RWO and performs the initialization
  • All other pods mount a pvc as ROX (ReadOnlyMany) with 1st pvc as datasource
  • Or each pod can have its own rwo pvc
• HA deployments may need additional considerations

Recommend

• Update ca-bundle secret using something like external secrets, this way if the bundle is updated, pods will be updated at next start (or when skater restarts them).

Additional

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ca-bundle-server
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ca-bundle-reposerver
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Mi

initContainers

  server:
    initContainers:
    - name: get-ca-bundle
      image: docker.io/alpine/curl:latest
      command:
      - "/bin/sh"
      - "-c"
      args:
      - |
        for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
        /usr/sbin/update-ca-certificates
        rm -rf /tmp/target/*
        cp -r /etc/ssl/certs/* /tmp/target/
      volumeMounts:
      - name: ca-bundle
        mountPath: /tmp/source
      - name: pvc-ca-bundle
        mountPath: /tmp/target
    volumes:
    - name: ca-bundle
      secret:
        secretName: ca-bundle
    - name: pvc-ca-bundle
      persistentVolumeClaim:
        claimName: ca-bundle-server

    volumeMounts:
    - name: pvc-ca-bundle
      mountPath: "/etc/ssl/certs"

  repoServer:
    initContainers:
    - name: get-ca-bundle
      image: docker.io/alpine/curl:latest
      command:
      - "/bin/sh"
      - "-c"
      args:
      - |
        for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
        /usr/sbin/update-ca-certificates
        rm -rf /tmp/target/*
        cp -r /etc/ssl/certs/* /tmp/target/
      volumeMounts:
      - name: ca-bundle
        mountPath: /tmp/source
      - name: pvc-ca-bundle
        mountPath: /tmp/target
    volumes:
    - name: ca-bundle
      secret:
        secretName: ca-bundle
    - name: pvc-ca-bundle
      persistentVolumeClaim:
        claimName: ca-bundle-reposerver

    volumeMounts:
    - name: pvc-ca-bundle
      mountPath: "/etc/ssl/certs"

[lknite]

The idea to just mount a configmap /secret should not be considered a solution. This works in most cases but requires a person to start googling until they find out the right location to put the file vs just searching for 'bundle' or 'cert' in the values.yaml and being done. Why not just let the user specify a secret to use and put it where it needs to go ... the consumer shouldn't have to know what language or image file is being used to get the onprem certs installed.

[hamelg]

Another workaround with the operator by specifying volumes and volumeMounts in the crd argocds.argoproj.io, ca certificats are in the configmap ca-certs :

  repo:
    volumeMounts:
      - mountPath: /etc/ssl/certs/ca-bundle-private.crt
        name: ca-certs
        subPath: ca-bundle.crt
    volumes:
      - configMap:
          name: ca-certs
        name: ca-certs

[jmanuelortizn]

I am facing the same issue when pulling docker images from a custom private registry with self signed cert.

• I tried the proposed solution with mounted volumes but the same issue is still happening: (I am able to see the cusom certs in /etc/ssl/certs/* and also run update-ca-certificates)

Failed to pull image "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
rpc error: code = Unknown desc = failed to pull and unpack image "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
failed to resolve reference "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
failed to do request: Head "https://docker.pvt-registry.net/v2/image-service/manifests/2.0.0.76-develop": 
x509: certificate signed by unknown authority

Is there other work around that worked for anyone?

[jle-pass]

+1