Open lknite opened 1 month ago
@lknite are you using Dex?
@krancour no, am using keycloak
And the failure is occurring right when the API server starts?
yes, the pod never becomes ready cause it detects the cert cannot be verified during startup
Ok. Got it. I am thinking back to when we first did the ODIC integration and trying to recall why I wouldn't have made provisions for a custom CA bundle right from the get go...
My vague recollection is that the possibility of identity providers without a cert that is trusted by users' browsers hadn't seemed realistic. After all, if one doesn't trust the IDP's cert without using a custom CA bundle, neither does the other.
Are your users installing a custom CA bundle into their browsers? Or are they just clicking through the warnings?
I guess what it all boils down to was an assumption of (with the possible exception of the included Dex) IDPs being properly secured with widely-trusted certificates. (An IDP seems like the exact sort of thing where you're not going to cut corners on security, after all.)
I'm open to that assumption being challenged. Can you tell me more about your use case?
Vault is used via cert-manager to generate certificates within all clusters. The ca-bundle root certificates and intermediate certificates are pushed out to client systems automatically so that clients can experience cert validation with applications running via kubernetes.
Tanzu is used to stand up kubernetes clusters, and there is a place to specify the vault ca-bundle so that it also reaches workernodes of kubernetes clusters.
However, applications running in kubernetes still need a ca-bundle mounted to /etc/ssl/certs or wherever is appropriate within a pod to get it to trust the oidc provider or any other kubernetes-based app. I've currently hacked together a solution to mount a configmap to /etc/ssl/certs using kyverno, but given that no one really knows ahead of time where the correct place to mount a ca-bundle is in any given container, I certainly always appreciate it when the option is available via a helm chart. (I wouldn't expect everyone to have kyverno experience in order to be able to deploy kargo. Even myself, I didn't have kyverno experience until last week when I first set it up, thank goodness so I could use it today.)
The keycloak IDP that I'm using is onprem without a publicly resolvable cert, it's running in kubernetes, pretty much everything does.
This is at work, however I plan to also stand up and use kargo at my house for home projects, I'll also have an onprem keycloak there that I only use internally, and will need to similarly mount a ca-bundle. I've had to mount a ca-bundle for every app I've setup to use keycloak, some provide a way to specify the cert, others just give the option to mount a volumeMount and you get to figure out where you are supposed to mount it.
There is another way to "resolve" this that you might prefer, 'harbor' has an option to just disable the verification of a tls cerficate with its OIDC configuration (see the checkbox at the bottom of this configuration gui) ... obviously not ideal, but I could see it existing for development purposes (though I'd rather specify a ca-bundle):
@lknite thanks for the details. I get how you're distributing CA bundles around to things in Kubernetes. But what happens with interactions between the user's browser and the IDP? There are basically three possibilities:
If it were no.1, Kargo should trust the IDP as much as the browser does. (So I assume it's not no.1.) If it were no.2, it feels dangerous to ignore cert warnings on an IDP. This is what lead to the (possibly poor) assumption that people don't do this. For this case, I'd recommend using a more widely trusted certificate if that is an option or else falling back to no.3.
No.3 seems to be the intersection of secure + plausible and is why I'm willing to be convinced to make it easier to install CA bundles.
looks like we are using your 3rd option, except that the users don't install the ca bundle themselves, we auto-install it to their computers so they see the certificate verified icon whenever accessing something running via kubernetes
Ok. Makes sense! Consider me convinced that it's not only people who have thrown caution to the wind (case 2) who would benefit from this.
I'll put some thought into how best to approach this.
Is it in the 0.8.1 helm chart and I just don't see it?
@lknite no. This hasn't been assigned to any milestone yet. In light of current internal priorities, a community contribution might be the fastest path to getting this done.
I used kyverno to mount the needed certs into /etc/ssl/certs ... things seemed ok until I tried my first git-based promotion then I saw this in the logs:
[kargo-controller-7c8c67bfb9-ntdzm] time="2024-08-05T00:34:51Z" level=error msg="error executing Promotion" error="error executing Git-based promotion mechanisms: error executing Kustomize promotion mechanism: error cloning git repo \"https://gitlab.com/lknite/lido-managers-deploy.git\": error cloning repo \"https://gitlab.com/lknite/lido-managers-deploy.git\" into \"/tmp/repo-2409851590/repo\": error executing cmd [/usr/bin/git clone --no-tags https://gitlab.com/lknite/lido-managers-deploy.git /tmp/repo-2409851590/repo]: Cloning into '/tmp/repo-2409851590/repo'...\nfatal: unable to access 'https://gitlab.com/lknite/lido-managers-deploy.git/': error setting certificate file: /etc/ssl/certs/ca-certificates.crt\n" freight=b87f75a60e71b2b45aa0d897fa21a49ebcfb9909 namespace=kargo-lido-managers promotion=non-prod.01j4fzbp6hwp8pc21abrxzkvrm.b87f75a stage=non-prod
[kargo-controller-7c8c67bfb9-ntdzm] time="2024-08-05T00:34:51Z" level=info msg=promotion freight=b87f75a60e71b2b45aa0d897fa21a49ebcfb9909 namespace=kargo-lido-managers phase="\"Errored\"" promotion=non-prod.01j4fzbp6hwp8pc21abrxzkvrm.b87f75a stage=non-prod
It doesn't like my work around which ended up making that path read-only:
error setting certificate file: /etc/ssl/certs/ca-certificates.crt\n"
I removed my /etc/ssl/certs mount and the controller can promote via git now ... but it fails when accessing my container repo:
[kargo-controller-6ffd54c5c4-z4j6b] time="2024-08-05T00:49:32Z" level=error msg="Reconciler error" Warehouse="{harbor kargo-lido-managers}" controller=warehouse controllerGroup=kargo.akuity.io controllerKind=Warehouse error="error discovering artifacts: error discovering images: error discovering latest images \"harbor.vc-prod.k.home.net/lido/brokermanager\": error discovering newest applicable images \"harbor.vc-prod.k.home.net/lido/brokermanager\": error listing tags: error listing tags for repo URL harbor.vc-prod.k.home.net/lido/brokermanager: Get \"https://harbor.vc-prod.k.home.net/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority" name=harbor namespace=kargo-lido-managers reconcileID="\"1effb429-787b-490a-b650-7612f6a96248\""
This works. Just tried it out. Probably would be good if we could support either a 'secret' or 'configmap' with the certs & personally I'd like to have the option to use a pvc instead of an emptydir. I hear emptydir is sometimes blocked by policy.
I'm forking the repo and taking a look.
spec:
containers:
- command:
- /usr/local/bin/kargo
- controller
envFrom:
- configMapRef:
name: kargo-controller
image: ghcr.io/akuity/kargo:v0.8.2
name: controller
volumeMounts:
- name: certs
mountPath: /etc/ssl/certs
initContainers:
- name: parse-ca-bundle
image: docker.io/alpine/curl:latest
command:
- "/bin/sh"
- "-c"
args:
- |
for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
/usr/sbin/update-ca-certificates
rm -rf /tmp/target/*
cp -r /etc/ssl/certs/* /tmp/target/
volumeMounts:
- name: ca-bundle
mountPath: /tmp/source
- name: certs
mountPath: /tmp/target
volumes:
- name: ca-bundle
configMap:
name: k.home.net
- name: certs
emptyDir: {}
Pull request created to allow a 'ca-bundle to be added to the 'controller': https://github.com/akuity/kargo/pull/2382
kyverno users can use the following policy as a work around in the meantime:
$ cat add-certificates-volume.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-certificates-volume
annotations:
policies.kyverno.io/title: Add Certificates as a Volume
policies.kyverno.io/category: Sample
policies.kyverno.io/subject: Pod,Volume
# kyverno.io/kyverno-version: 1.5.2
# kyverno.io/kubernetes-version: "1.21"
# policies.kyverno.io/minversion: 1.5.0
pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,Job,StatefulSet
policies.kyverno.io/description: >-
In some cases you would need to trust custom CA certificates for all the containers of a Pod.
It makes sense to be in a ConfigMap so that you can automount them by only setting an annotation.
This policy adds a volume to all containers in a Pod containing the certificate if the annotation
called `inject-certs` with value `enabled` is found.
spec:
# background: false
rules:
Checklist
kargo version
, if applicable.Proposed Feature
Motivation
Suggested Implementation
implement two methods which may be used: