Open raravena80 opened 5 years ago
@raravena80 I'm not aware of any certificate created by kubeadm under /var/lib/kubelet/pki/
.. could you kindly provide more info? e.g kubeadm config files, steps to create the cluster
@fabriziopandini I'm not entirely sure if the certs are created by the kubeadm, but the general procedure is described here.
This is what the content of the directory looks like:
root@ip-172-31-1-118:/var/lib/kubelet/pki# pwd
/var/lib/kubelet/pki
root@ip-172-31-1-118:/var/lib/kubelet/pki# ls -al
total 24
drwxr-xr-x 2 root root 4096 Jul 23 21:10 .
drwxr-xr-x 7 root root 4096 Nov 12 04:52 ..
-rw------- 1 root root 2810 Jul 23 21:09 kubelet-client-2018-07-23-21-09-53.pem
-rw------- 1 root root 1159 Jul 23 21:10 kubelet-client-2018-07-23-21-10-43.pem
lrwxrwxrwx 1 root root 59 Jul 23 21:10 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-07-23-21-10-43.pem
-rw-r--r-- 1 root root 1501 Nov 8 23:53 kubelet.crt
-rw------- 1 root root 1679 Nov 8 23:53 kubelet.key
root@ip-172-31-1-118:/var/lib/kubelet/pki#
Are they kubelet.crt
and kubelet.key
files created by the kubelet the first time it loads?
@raravena80 thanks for the clarification Probably I don't have the full context here so I leave room to others for answering.
Only one side note (might be it can help)
Kubeadm already creates a certificate named apiserver-kubelet-client
for letting the api server talking securely with the kubelets; it is signed by ca and bound to necessary RBAC rules.
/assign @liztio
I think this is to pre-generate kubelet's server certs. I tinkered with trying to use the Kubelet flags for TLS server bootstrap and rotate server certs, unfortunately I could not get Kubelet to request a server cert for itself using the bootstrap token. Kubelet ends up falling back to its default behavior for server certs, which is to generate a self-signed one.
To the best of my knowledge, right now the only way around that is to generate Kubelet's server certs out-of-band and place them at a deterministic path and kubelet (configured by kubeadm) will pick it up, and set some kubelet flags accordingly; reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates
The apiserver-kubelet-client
is the client cert that API server will present to a kubelet, but kubelet is configured to trust clients that are signed by the k8s CA:
# cat /var/lib/kubelet/config.yaml
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
It's the kubelet's identity as a server being presented that needs to be signed by the k8s CA, which comes around to the original question.
There's also some relevant discussion at the end of this thread: https://github.com/kubernetes/kubeadm/issues/118
I think kubeadm may have to add a CSR approver for server cert requests with a valid bootstrap token, just like it does for client cert requests?
What about having the kubelet upload its self signed ca to a configmap somewhere? the nodeadmission plugin could restrict it to just its own configmap. metrics-server can use it to contact the node.
Any ideas about that?
If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.
@alexbrand I agree on that
kubelet TLS bootstrapping only generates client certificates for whatever reason:
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
And kubeadm already does this. Perhaps this is a kubelet feature request?
lets summarize the issue:
as outlined by @anitgandhi: https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-454572577
the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:
--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key
without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:
sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt
with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt
), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname
.
possible solutions:
A) implement singing of a new kubelet.crt/key
pair, ideally under /var/lib/kubelet/pki
and setting the extra kubelet flags --tls-cert-file
, --tls-private-key-file
.
B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524 except possibly using Kubernetes CSRs / kubeadm commands.
C) as commented by @alexbrand
If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.
D) ?
@kubernetes/sig-cluster-lifecycle to me this seems in the space between bug/feature.
also see: https://github.com/kubernetes/community/pull/602/files
I think something between options B+C should be done since a lot of the bootstrap token client cert/CSR logic kubelet+kubeadm would have common logic to this.
lets summarize the issue:
as outlined by @anitgandhi: #1223 (comment)
the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:
--tls-cert-file=<some-path>/kubelet.crt --tls-private-key-file=<some-path>/kubelet.key
without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:
sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt
with a self-signed certificate instead of certificate signed by the cluster CA (
/etc/kubernetes/ca.crt
), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only includeDNS:hostname
.possible solutions: A) implement singing of a new
kubelet.crt/key
pair, ideally under/var/lib/kubelet/pki
and setting the extra kubelet flags--tls-cert-file
,--tls-private-key-file
.B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524 except possibly using Kubernetes CSRs / kubeadm commands.
C) as commented by @alexbrand
If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.
D) ?
@kubernetes/sig-cluster-lifecycle to me this seems in the space between bug/feature.
also see: https://github.com/kubernetes/community/pull/602/files
great summary @neolit123 . Do you know if this will slip onto next cycle or work in progress as we speak ? Asking mainly because of the metrics-server which imo every deployment wants to have it ;)
@randomvariable mentioned that there is another workaround for that. from the discussions thus far, we are hesitant about signing the kubelet-serving cert with the cluster CA. this topic needs further discussion.
/remove-help
because the solution to implement has not been chosen yet.
Any movement on this? I'm running up against it to support autoscaling features within a kubeadm deployed cluster.
Current workaround is to turn off CA checking of the kubelet certificate.
helm install --set 'args={--kubelet-insecure-tls}' --namespace kube-system metrics stable/metrics-serve
not really, it's blocked on design proposals. there are a number of workarounds but the work of documenting those stalled: https://github.com/kubernetes/kubeadm/issues/1602
--kubelet-insecure-tls
this might not be ideal for all users.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
/lifecycle frozen
Running into this exact issue creating v1.18.2 cluster with kubeadm.
When setting up metrics-server it doesn't work without either setting it's kubelet-insecure-tls
flag OR issue certificates for kublet "out of band", signing it with the kubernetes CA.
I thought about re-using the kubelet client certificate but that's of course issued to CN = system:node:nodename
and no SANs. And I did test it though wich ofcourse changes the error to indicate just that. The same certificate could be used as both server/client if it had the nodename as a subject alternate name? But I'm guessing it'd be more proper to use separate certs for server/client?
/remove-lifecycle frozen
/lifecycle frozen
it's frozen so that the bots don't close the issue.
The same certificate could be used as both server/client if it had the nodename as a subject alternate name?
in theory and unless the kubelet validates them - e.g. "client cert must not have SANs".
But I'm guessing it'd be more proper to use separate certs for server/client?
it is the common practice to use them separate even in cases where it seems avoidable. it's unlikely that the kubelet/auth{z|n} maintainers would change this detail.
Hey. Did a little more diggin. Kubelet configuration option serverTLSBootstrap: true
can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?
Setting both rotateCertificates: true
and serverTLSBootsrap: true
and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is for O = system:nodes, CN = system:node:<nodename>
with Subject Alternate Names for DNS: <nodename>, IP Address: <node IP address>
Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?
Hey. Did a little more diggin. Kubelet configuration option
serverTLSBootstrap: true
can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?Setting both
rotateCertificates: true
andserverTLSBootsrap: true
and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is forO = system:nodes, CN = system:node:<nodename>
with Subject Alternate Names forDNS: <nodename>, IP Address: <node IP address>
Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?
Not sure the security implementations but you can combine serverTLSBootstrap
with this operator to auto-approve the CSRs https://github.com/kontena/kubelet-rubber-stamp
Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?
kubeadm cannot do the approval because kubeadm is not a daemon. it has to deploy a controller / operator that manages that for the user. maybe in the future.
the certificates API should go GA soon and hopefully we are going to have a better way to manage this in k8s. please watch: https://github.com/kubernetes/enhancements/issues/267 (yet unclear to me what we are going to end up with...)
we also have alternatives ideas. but if all this is trying to solve the metric-server issue, you might as well just use https://github.com/brancz/kube-rbac-proxy that can perform SAR on the MS requests to the kubelet . sadly this is not yet documented on our side: https://github.com/kubernetes/kubeadm/issues/1602
@neolit123 I at least started looking into it when trying to stand up metrics server on kubeadm and "the hard way" clusters for learning experience. Easiest way there was of course to flag MS with --kubelet-insecure-tls
, but I really wanted to see how to fix it in a secure way and then just got interested in the issue. 🙂
For now it's easy enough for me to add the serverTLSbootstrap
flag to the kubelet config and manually approve the certificates. I have noticed a downside of it though, which is that you can't fully interact with pods on the node until you approve the cert. (kubectl exec fails to run command on pods running on a node before approval for example)
I'll follow along the enhancements issue too. Thanks.
That's really sad that with kubeadm which seems mature enough the out of the box result for kubeletet cert is to be self signed & many people choose kubelet-insecure-tls
for metrics server instead of doing things properly & etc :(
it's a complicated problem.
please try: https://github.com/kontena/kubelet-rubber-stamp or https://github.com/brancz/kube-rbac-proxy as workarounds
it's a complicated problem.
please try: https://github.com/kontena/kubelet-rubber-stamp or https://github.com/brancz/kube-rbac-proxy as workarounds
Actually https://github.com/kontena/kubelet-rubber-stamp works pretty good & imo seems to be a more correct solution instead of proxy.
Step 1:
Add
serverTLSBootstrap: true
to the end of every /var/lib/kubelet/config.yaml
for kubelets reconfiguration & don't forget to apply config (or just reboot them)
Step 2: Deploy kubelet-rubber-stamp
service_account.yaml
role.yaml
role_binding.yaml
operator.yaml
Step 3:
Edit metrics-server deployment & remove --kubelet-insecure-tls
Result:
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-7dvsx 31m kubernetes.io/kubelet-serving system:node:u-02 Approved,Issued
csr-d6rvm 31m kubernetes.io/kubelet-serving system:node:u-03 Approved,Issued
csr-szblz 31m kubernetes.io/kubelet-serving system:node:u-01 Approved,Issued
csr-zjfgj 31m kubernetes.io/kubelet-serving system:node:u-04 Approved,Issued
Hey, just to add to that @vainkop
During your initial kubeadm init
to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set the serverTLSBootstrap
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
...
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
kubeadm init --config=kubeadm-config.yaml
Then all kubelet's will automatically be set up using the serverTLSBootstrap
flag.
To get the CSRs
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-2qkdw 2m1s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:fcufbo Approved,Issued
csr-9wvgt 114s kubernetes.io/kubelet-serving system:node:worker-1 Pending
csr-lz97v 4m58s kubernetes.io/kubelet-serving system:node:master-1 Pending
csr-rsdsp 4m59s kubernetes.io/kube-apiserver-client-kubelet system:node:master-1 Approved,Issued
csr-wgxqs 4m49s kubernetes.io/kubelet-serving system:node:master-1 Pending
Then either approve them manually or deploy https://github.com/kontena/kubelet-rubber-stamp which approves them automatically. I just tried it with kubelet-rubber-stamp and it works great.
Also I did not seem to need to restart the kubelet's this way, they picked up their certificates as soon as I approvde the CSR, but a caveat is that the kublet's have NO cert until the CSR is approved, it does not get a self signed certificate first.
kubectl certificate approve csr-ab123 # OR deploy rubber-stamp!
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-9wvgt 3m kubernetes.io/kubelet-serving system:node:worker-1 Approved,Issued
...
Another strange thing seems to happen here btw, which is that the master node seems to create it’s CSR twice. (At least the two times I tried this)
But as @nijave says in a comment above, I'm not sure what the security implications of using the rubber-stamper are.
@allir, @vainkop as far is i can see the kubelet-rubber-stamp does only verify if the common name of the CSR matches the requester name but does not verify whether the additional hostnames and IP Addresses requested by the kubelet are valid. This means an attacker that has access to the kubelet client certificate can create certificates for basically any domain name or IP address. All clients that are configured to trust the root CA will then accept this certificate. Of course validating the hostname and IP addresses that are valid for a given kubelet is hard since there is currently no authority that can confirm what a kubelet is allowed to request. For example using the node object on the API server is not sufficient because kubelets can update the object without limits.
Hey, just to add to that @vainkop During your initial
kubeadm init
to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set theserverTLSBootstrap
kubeadm init --config=kubeadm-config.yaml
Then all kubelet's will automatically be set up using theserverTLSBootstrap
flag.
Or for an existing K8s setup using Ansible it can be:
tasks:
- name: Insert a line at the end of /var/lib/kubelet/config.yaml
lineinfile:
path: /var/lib/kubelet/config.yaml
line: 'serverTLSBootstrap: true'
+restart kubelets
Wow I'm so glad that I found this issue, and I'm not alone who wants to make this proper way. :)
Now let me share my thoughts on this issue (please correct me if I'm somewhere wrong):
First my vision of the original problem:
Currently kubeadm enables webhook authentication for all kubelets by default, so kubelet is validating client certificates for incoming connections with no problem even if --kubelet-insecure-tls
option is specified.
From the other side metrics-server have no oportunity to verify specific kubelet certificate because it is self-signed on the node.
Possible risks of using --kubelet-insecure-tls
for metrics-server:
While the kubelet data is somewhat secured and will never be provided to the metrics-server without successful webhook authentification.
In theory someone can compromise the server IP or hostname and provide wrong statistics. But for establishing the connections metricserver is using an IP address and hostnames specified for node via kube-apiserver, so the attacker needs to hack the API-server, DNS or the node IP-address first.
Little observation:
The metrics-server is not single service which is accessing the kubelets directly. Kube-apiserver also doing this for read the container logs or execute the shell on them. The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate?
Doesn't it behave the same as the metrics server with the --kubelet-insecure-tls
option in this case?
Possible solution: Nowdays the webhooks and API aggregation are quite popular in Kubernetes. All of them are behave the similar way, by generating it's own CA and crt/key pair. The CA hash is also stored in a specific resource to provide kube-apiserver with information about which certificate it can trust.
For example:
APIServices are storing related CA hash in their apiservices.apiregistration.k8s.io
resource:
spec:
caBundle: <ca-hash>
Webhooks are storing related CA hash in their validatingwebhookconfigurations.admissionregistration.k8s.io
and mutatingwebhookconfigurations.admissionregistration.k8s.io
resources:
webhooks:
- clientConfig:
caBundle: <ca-hash>
To me it is pretty obvious that each node resource should have similar caBundle
in their spec
, where kubelets can register their own CA for serving using their client certificate:
spec:
caBundle: <ca-hash>
Both metris-server and kube-apiserver should use these certificates to verify and trust the connection to the kubelets.
thanks to @kfox1111 who expressed a similar idea previously https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-460854312
The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate? Doesn't it behave the same as the metrics server with the
--kubelet-insecure-tls
option in this case?
To answer this question I can quote @luxas here:
Right, we can't make the connections from the api server to the kubelet servers verified, as each kubelet has its own self-signed cert. We might consider a manual approving flow wrt kubelet serving certs in the future but that's not secured by default at the moment.
from https://github.com/kubernetes/kubeadm/issues/118#issuecomment-407498529
hope it can be resolved some day
[root@jenkins metrics-server]# kubectl -n kube-system logs -f metrics-server-6955d88db9-lftlz
I1120 08:23:09.094132 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1120 08:23:09.094234 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1120 08:23:09.094270 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094279 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094307 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.094315 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.095064 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:09.095207 1 secure_serving.go:197] Serving securely on [::]:4443
I1120 08:23:09.095259 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1120 08:23:09.194453 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.194660 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.194455 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
E1120 08:23:10.420643 1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node k8s-master3: unable to fetch metrics from node k8s-master3: Get "https://10.39.140.250:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.250 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master1: unable to fetch metrics from node k8s-master1: Get "https://10.39.140.248:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.248 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master2: unable to fetch metrics from node k8s-master2: Get "https://10.39.140.249:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.249 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node1: unable to fetch metrics from node k8s-node1: Get "https://10.39.140.251:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.251 because it doesn't contain any IP SANs]
I1120 08:23:33.874949 1 requestheader_controller.go:183] Shutting down RequestHeaderAuthRequestController
I1120 08:23:33.874978 1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:33.874993 1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:33.875019 1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
I1120 08:23:33.875026 1 dynamic_serving_content.go:145] Shutting down serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:33.875041 1 secure_serving.go:241] Stopped listening on [::]:4443
No news from this issue? I would be interested to have a solution for this as well.
we are documenting workarounds here: https://github.com/kubernetes/website/pull/27071 https://github.com/kubernetes/kubeadm/issues/1602
we can keep this issue open, but due to the complexities of requiring to deploy a signer with kubeadm by default, it's unlikely that we are going to make this change anytime soon.
This would mean keeping track of the previous state. Maybe it can work. You can talk with the maintainers about this idea in #sig-auth on k8s slack. On Oct 6, 2021 12:08, "DrmagicE" @.***> wrote:
How about make the kube-controller can auto-approve those CSRs that are not changing anything or not changing some sensitive field such as IP address, DNS name, common name, organization etc.? I think in most cases, we just want to renew the certificate without any changes. It would be nice if the kube-controller can auto-approve those CSRs.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-935802715, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACRATDGFQUXQWCV4XE2HIDUFQGZPANCNFSM4GCXG3JQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
we are documenting workarounds here: kubernetes/website#27071 #1602
we can keep this issue open, but due to the complexities of requiring to deploy a signer with kubeadm by default, it's unlikely that we are going to make this change anytime soon.
Can someone comment on what are possible consequences if we would use serverTLSBootstrap: true
configuration as a workaround? Does it enable some other processes we should know about? Or better yet, why is this setting not enabled by default?
It would help us to understand possible risks and decide if enabling this configuration is somewhat better than running metrics-server with --kubelet-insecure-tls
(we basically need it only for metrics-server).
Some kind of integration with spire would be interesting. There is a systemd workload attestor now, so targeting a cert to a kubelet via systemd kubelet service would be much easier to do now.
I also faced the insecure kubelet server certs and went down the same path as most of the comments here. Setting serverTLSBootstrap: true
and manually approving the csr seems like the way to go for now.
I first thought you could set up a auto approver via kube-controller with the the needed RBAC permissions but the kube controller will nver auto approve the kubelet server certs, just client configurations.
Having an optional (opt-in) aption to also allow auto approve for kubelet server certs might me another option to avoid people using --insecure
I also faced the insecure kubelet server certs and went down the same path as most of the comments here. Setting
serverTLSBootstrap: true
and manually approving the csr seems like the way to go for now.I first thought you could set up a auto approver via kube-controller with the the needed RBAC permissions but the kube controller will nver auto approve the kubelet server certs, just client configurations.
Having an optional (opt-in) aption to also allow auto approve for kubelet server certs might me another option to avoid people using
--insecure
There are 3rd party options like https://github.com/postfinance/kubelet-csr-approver
Is this a BUG REPORT or FEATURE REQUEST?
/kind bug
Opening the
kubeadm
side for this issue on the metrics-serverVersions
Environment:
kubectl version
):uname -a
):What happened?
kubeadm creates certs under
/var/lib/kubelet/pki/kubelet.*
signed with a different CA from the one under/etc/kubernetes/pki/ca.pem
What you expected to happen?
As a result some apps like the metrics-server cannot collect stats from a secured kubelet because the kubelet has certs signed by a different ca from the K8s master(s)
Error sample:
How to reproduce it (as minimally and precisely as possible)?
Install the metrics-server on run:
$ kubectl -n kube-system logs
Anything else we need to know?
Some more background here
There also steps in there that I followed to fix the issue.
edit: neolit123
the problem here is that the serving cert is self-signed by default: see https://github.com/kubernetes/website/pull/27071 for documentation update.