oidc server specified, certificate required to validate oidc server, where to specify certificate in helm chart?

[lknite]

Checklist

• [x] I've searched the issue queue to verify this is not a duplicate feature request.
• [x] I've pasted the output of kargo version, if applicable.
• [x] I've pasted logs, if applicable.

Proposed Feature

• ability to specify certificate needed to verify oidc server via helm chart

Motivation

• after specifying oidc server, client_id, client_secret, etc ... unable to login because certificate verification fails

Suggested Implementation

implement two methods which may be used:

1. allow a ca-bundle to be provided via a secret:
   • api.oidc.caBundle.secretName
   • api.oidc.caBundle.key
2. allow a ca-bundle to be provided in-line
   • api.oidc.caBundle
   • allow the ca-bundle to be one or more certificates

[krancour]

@lknite are you using Dex?

[lknite]

@krancour no, am using keycloak

[krancour]

And the failure is occurring right when the API server starts?

[lknite]

yes, the pod never becomes ready cause it detects the cert cannot be verified during startup

[krancour]

Ok. Got it. I am thinking back to when we first did the ODIC integration and trying to recall why I wouldn't have made provisions for a custom CA bundle right from the get go...

My vague recollection is that the possibility of identity providers without a cert that is trusted by users' browsers hadn't seemed realistic. After all, if one doesn't trust the IDP's cert without using a custom CA bundle, neither does the other.

Are your users installing a custom CA bundle into their browsers? Or are they just clicking through the warnings?

I guess what it all boils down to was an assumption of (with the possible exception of the included Dex) IDPs being properly secured with widely-trusted certificates. (An IDP seems like the exact sort of thing where you're not going to cut corners on security, after all.)

I'm open to that assumption being challenged. Can you tell me more about your use case?

[lknite]

Vault is used via cert-manager to generate certificates within all clusters. The ca-bundle root certificates and intermediate certificates are pushed out to client systems automatically so that clients can experience cert validation with applications running via kubernetes.

Tanzu is used to stand up kubernetes clusters, and there is a place to specify the vault ca-bundle so that it also reaches workernodes of kubernetes clusters.

However, applications running in kubernetes still need a ca-bundle mounted to /etc/ssl/certs or wherever is appropriate within a pod to get it to trust the oidc provider or any other kubernetes-based app. I've currently hacked together a solution to mount a configmap to /etc/ssl/certs using kyverno, but given that no one really knows ahead of time where the correct place to mount a ca-bundle is in any given container, I certainly always appreciate it when the option is available via a helm chart. (I wouldn't expect everyone to have kyverno experience in order to be able to deploy kargo. Even myself, I didn't have kyverno experience until last week when I first set it up, thank goodness so I could use it today.)

The keycloak IDP that I'm using is onprem without a publicly resolvable cert, it's running in kubernetes, pretty much everything does.

This is at work, however I plan to also stand up and use kargo at my house for home projects, I'll also have an onprem keycloak there that I only use internally, and will need to similarly mount a ca-bundle. I've had to mount a ca-bundle for every app I've setup to use keycloak, some provide a way to specify the cert, others just give the option to mount a volumeMount and you get to figure out where you are supposed to mount it.

[lknite]

There is another way to "resolve" this that you might prefer, 'harbor' has an option to just disable the verification of a tls cerficate with its OIDC configuration (see the checkbox at the bottom of this configuration gui) ... obviously not ideal, but I could see it existing for development purposes (though I'd rather specify a ca-bundle):

[krancour]

@lknite thanks for the details. I get how you're distributing CA bundles around to things in Kubernetes. But what happens with interactions between the user's browser and the IDP? There are basically three possibilities:

1. Your IDP uses a cert trusted by the browser's default CA bundle.
2. Your IDP uses a cert not trusted by the browser's default CA bundle and users ignore the warnings their browser raises.
3. Your IDP uses a cert not trusted by the browser's default CA bundle, but your users all install a CA bundle that trusts the IDP's cert.

If it were no.1, Kargo should trust the IDP as much as the browser does. (So I assume it's not no.1.) If it were no.2, it feels dangerous to ignore cert warnings on an IDP. This is what lead to the (possibly poor) assumption that people don't do this. For this case, I'd recommend using a more widely trusted certificate if that is an option or else falling back to no.3.

No.3 seems to be the intersection of secure + plausible and is why I'm willing to be convinced to make it easier to install CA bundles.

[lknite]

looks like we are using your 3rd option, except that the users don't install the ca bundle themselves, we auto-install it to their computers so they see the certificate verified icon whenever accessing something running via kubernetes

[krancour]

Ok. Makes sense! Consider me convinced that it's not only people who have thrown caution to the wind (case 2) who would benefit from this.

I'll put some thought into how best to approach this.

[lknite]

Is it in the 0.8.1 helm chart and I just don't see it?

[krancour]

@lknite no. This hasn't been assigned to any milestone yet. In light of current internal priorities, a community contribution might be the fastest path to getting this done.

[lknite]

I used kyverno to mount the needed certs into /etc/ssl/certs ... things seemed ok until I tried my first git-based promotion then I saw this in the logs:

[kargo-controller-7c8c67bfb9-ntdzm] time="2024-08-05T00:34:51Z" level=error msg="error executing Promotion" error="error executing Git-based promotion mechanisms: error executing Kustomize promotion mechanism: error cloning git repo \"https://gitlab.com/lknite/lido-managers-deploy.git\": error cloning repo \"https://gitlab.com/lknite/lido-managers-deploy.git\" into \"/tmp/repo-2409851590/repo\": error executing cmd [/usr/bin/git clone --no-tags https://gitlab.com/lknite/lido-managers-deploy.git /tmp/repo-2409851590/repo]: Cloning into '/tmp/repo-2409851590/repo'...\nfatal: unable to access 'https://gitlab.com/lknite/lido-managers-deploy.git/': error setting certificate file: /etc/ssl/certs/ca-certificates.crt\n" freight=b87f75a60e71b2b45aa0d897fa21a49ebcfb9909 namespace=kargo-lido-managers promotion=non-prod.01j4fzbp6hwp8pc21abrxzkvrm.b87f75a stage=non-prod
[kargo-controller-7c8c67bfb9-ntdzm] time="2024-08-05T00:34:51Z" level=info msg=promotion freight=b87f75a60e71b2b45aa0d897fa21a49ebcfb9909 namespace=kargo-lido-managers phase="\"Errored\"" promotion=non-prod.01j4fzbp6hwp8pc21abrxzkvrm.b87f75a stage=non-prod

It doesn't like my work around which ended up making that path read-only:

error setting certificate file: /etc/ssl/certs/ca-certificates.crt\n"

I removed my /etc/ssl/certs mount and the controller can promote via git now ... but it fails when accessing my container repo:

[kargo-controller-6ffd54c5c4-z4j6b] time="2024-08-05T00:49:32Z" level=error msg="Reconciler error" Warehouse="{harbor kargo-lido-managers}" controller=warehouse controllerGroup=kargo.akuity.io controllerKind=Warehouse error="error discovering artifacts: error discovering images: error discovering latest images \"harbor.vc-prod.k.home.net/lido/brokermanager\": error discovering newest applicable images \"harbor.vc-prod.k.home.net/lido/brokermanager\": error listing tags: error listing tags for repo URL harbor.vc-prod.k.home.net/lido/brokermanager: Get \"https://harbor.vc-prod.k.home.net/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority" name=harbor namespace=kargo-lido-managers reconcileID="\"1effb429-787b-490a-b650-7612f6a96248\""

[lknite]

This works. Just tried it out. Probably would be good if we could support either a 'secret' or 'configmap' with the certs & personally I'd like to have the option to use a pvc instead of an emptydir. I hear emptydir is sometimes blocked by policy.

I'm forking the repo and taking a look.

    spec:
      containers:
      - command:
        - /usr/local/bin/kargo
        - controller
        envFrom:
        - configMapRef:
            name: kargo-controller
        image: ghcr.io/akuity/kargo:v0.8.2
        name: controller
        volumeMounts:
        - name: certs
          mountPath: /etc/ssl/certs
      initContainers:
      - name: parse-ca-bundle
        image: docker.io/alpine/curl:latest
        command:
        - "/bin/sh"
        - "-c"
        args:
        - |
          for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
          /usr/sbin/update-ca-certificates
          rm -rf /tmp/target/*
          cp -r /etc/ssl/certs/* /tmp/target/
        volumeMounts:
        - name: ca-bundle
          mountPath: /tmp/source
        - name: certs
          mountPath: /tmp/target
      volumes:
      - name: ca-bundle
        configMap:
          name: k.home.net
      - name: certs
        emptyDir: {}

[lknite]

Pull request created to allow a 'ca-bundle to be added to the 'controller': https://github.com/akuity/kargo/pull/2382

[lknite]

quick workaround

kyverno users can use the following policy as a work around in the meantime:

• expects a configmap named 'cabundle'
• add 'inject-certs' annotation to each deployment via helm values:
  • kargo.api.annotations.inject-certs: "enabled"
  • kargo.controller.annotations.inject-certs: "enabled"
  • kargo.managementController.annotations.inject-certs: "enabled"

    
    $ cat add-certificates-volume.yaml 
    apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    metadata:
    name: add-certificates-volume
    annotations:
    policies.kyverno.io/title: Add Certificates as a Volume
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Pod,Volume
    #    kyverno.io/kyverno-version: 1.5.2
    #    kyverno.io/kubernetes-version: "1.21"
    #    policies.kyverno.io/minversion: 1.5.0
    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,Job,StatefulSet
    policies.kyverno.io/description: >-
    In some cases you would need to trust custom CA certificates for all the containers of a Pod.
    It makes sense to be in a ConfigMap so that you can automount them by only setting an annotation.
    This policy adds a volume to all containers in a Pod containing the certificate if the annotation
    called `inject-certs` with value `enabled` is found.      
    spec:
    #  background: false
    rules:

  • name: add-ssl-certs match: resources: kinds:
    • Deployment preconditions: all:
    • key: "{{ request.object.metadata.annotations.\"inject-certs\" || \"\" }}" operator: Equals value: enabled
    • key: "{{ request.operation || 'BACKGROUND'}}" operator: AnyIn value:
      • CREATE
      • UPDATE mutate: foreach:
    • list: "request.object.spec.template.spec.containers" patchStrategicMerge: spec: template: spec: containers:
      • name: "{{ element.name }}" volumeMounts:
        • name: certs mountPath: /etc/ssl/certs initContainers:
      • name: parse-cabundle image: docker.io/alpine/curl:latest command:
        • "/bin/sh"
        • "-c" args:
        • | for FILE in /tmp/source/; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done /usr/sbin/update-ca-certificates rm -rf /tmp/target/ cp -r /etc/ssl/certs/* /tmp/target/ volumeMounts:
        • name: cabundle mountPath: /tmp/source
        • name: certs mountPath: /tmp/target volumes:
      • name: cabundle configMap: name: cabundle
      • name: certs emptyDir: {}