Closed mjudeikis closed 5 months ago
I think the issue here is admin.kubeconfig
is generated for every shard by the KCP server - now that we're doing sharded deployments that won't make sense, because we'll end up with N admin.kubeconfig
files each with a shard specific kcp-admin bearer token.
I suspect the long term fix here is to remove the kubeconfig generation from KCP completely, but as a stopgap we may be able to have that point to the shard BaseURL instead of ExternalAddress.
It's currently possible to manually create a kubeconfig with the front-proxy CA and credentials of your choice (e.g client-cert, token, OIDC etc), but we don't currently document how to do that - we could add that to the README in this repo, or another alternative would be to create an additional kubeconfig via the chart (although in that case it should be optional and default to off, since in most non-dev cases the admin client won't run in the deployed KCP containers)
I suspect the long term fix here is to remove the kubeconfig generation from KCP completely, but as a stopgap we may be able to have that point to the shard BaseURL instead of ExternalAddress.
We won't delete that. Those kubeconfigs are meant as shard configs, and have system:master
membership. They are just not supposed to be exposed beyond one shard. They are intentionally not working with the proxy either.
Creating an admin kubeconfig out of band is the right approach IMO. There is only so much you can do in-place from within a chart.
They are just not supposed to be exposed beyond one shard
But they are currently pointing to frontProxy. Should we start exposing shards as public URLS: shard1.eu.faros.sh
shard2.eu.faros.sh
and creating kubeconfig for those? These would not be frontPrpxy routes but more as public endpoints for global FrontProxy?
I don't see "global scale" possible with all shards being in the same network? Or idea is that all shards are in same private network of some sort?
There should not be a need to talk to shards directly, with the exception of the virtual workspace URL (part of Shard.spec
; can be behind some load balancer). Hence, I would not encourage exposing shards directly.
Or idea is that all shards are in same private network of some sort?
Yes, that's the idea.
We won't delete that. Those kubeconfigs are meant as shard configs, and have
system:master
membership. They are just not supposed to be exposed beyond one shard. They are intentionally not working with the proxy either.
Ack, I was thinking that in a production environment you'd want to avoid non-expiring admin bearer tokens (even if they are only for internal use), and instead generate client certs which can be managed outside of KCP. I guess we can make the internal bearer-token stuff optional in future if needed.
Creating an admin kubeconfig out of band is the right approach IMO. There is only so much you can do in-place from within a chart.
I can work on a PR which shows how to do that in the README.
This is more of certificate bundles now. Need extend cert bundles to be more usable so users can add their own certs (Letencrypt including ) if they want to
will raise separate issue and close this one
I'm deploying KCP using a slightly modified chart version (https://github.com/mjudeikis/helm-charts/tree/alliasing). Most of the changes are indeed to workaround current limitations of the chart, like:
initContainer
for KCP to fix DigitalOcean storage limitationsValues.yaml file looks like bellow. It uses Nginx Passthrough as Ingress (replicate OpenShift router passthrough), and DNS in-place resolver for LetsEncrypt.
It uses Lets Encrypt for FrontProxy
kcp.faros.sh
certificates. The current certificate flow is :FrontProxy (Lets Enrypt) -> Shard (KCP-CA self signed Cert-manager managed)
from inside KCP shard pod:
Because of URL and certificate-authority-data mismatch:
admin.kubeconfig
is generated with FrontProxy URL and Shard certificate, and this breaks the Certificate trust chain as KCP-CA is not in the FrontProxy CA trust.For this to work
admin.kubeconfig
of the shard should be pointing toshard
itself, and go via FrontProxy OR it should have Let's Encrypt (or other CA authority used in FP deployment) incertificate-authority-data
.Workaround:
443
to6443
. Port must be 443 so DNS override would work.With this traffic inteded to kcp.faros.sh frontProxy goes directrly to shard endpoint, and things just work. But this is workaround around the fact that the
admin.kubeconfig
certificate and URL do not match.