Closed ne1000 closed 4 months ago
brancz
commented
6 years ago This is actually about the certificates that clients use to communicate with the Kubernetes API. Check the certificates that the kubelets, scheduler(s) and controller-manager(s) use.
ne1000
commented
6 years ago @brancz Yes, I did checked, but I didn't find something issue, kindly please give me a advise
# cat /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS="--address=0.0.0.0 \
--master=http://192.168.2.86:8080 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--leader-elect=true \
--v=0"
# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/doc
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --logtostderr=false --log-dir=/var/log/kubernetes --v=0 --cluster-dns=10.100.0.100 --cluster-domain=cluster.local. --resolv-conf=/etc/resolv.conf --authentication-token-webhook=true --authorization-mode=Webhook
Restart=on-failure
[Install]
WantedBy=multi-user.target
# cat /etc/kubernetes/kubelet.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
server: https://192.168.2.93:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: admin
name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate: /etc/kubernetes/ssl/admin.pem
client-key: /etc/kubernetes/ssl/admin-key.pem# cat /etc/kubernetes/bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
server: https://192.168.2.93:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet-bootstrap
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: d2b9e107b99641a01ff18e952cf9ce85# openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=izuf68thdbm0n4j5qywd7sz-ca@1533727394
Validity
Not Before: Aug 8 11:23:14 2018 GMT
Not After : Aug 8 11:23:14 2019 GMT
Subject: CN=izuf68thdbm0n4j5qywd7sz@1533727394
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d1:51:90:4a:e1:e5:e0:4c:90:f2:ff:ad:31:20:
77:34:d0:1a:7f:ab:c8:f5:87:74:10:4b:df:52:6a:
77:d7:01:92:ab:7a:14:4a:78:eb:c3:a7:9a:ed:f2:
b4:95:a7:dd:b8:40:25:4f:fb:06:d8:36:ef:4c:4b:
a9:13:0f:c9:f0:de:8a:f6:9a:17:1c:7c:07:5f:2f:
4a:dd:3c:f7:4e:7f:59:78:7b:0f:10:df:77:cc:bb:
1b:7f:02:3b:39:66:56:5c:37:3b:db:ec:c8:84:53:
46:ed:7e:26:3d:14:56:2d:f4:82:a3:4b:64:ae:8b:
3e:9c:56:c7:15:59:97:01:f7:93:6a:35:88:5d:b5:
cd:a5:03:02:0f:55:04:aa:77:6a:65:8e:96:2c:ae:
a6:7e:03:de:01:95:30:bc:68:21:52:4e:02:f4:c0:
ad:8f:6b:71:db:5b:b9:d3:7c:55:93:b1:ce:df:12:
be:1a:7e:95:0f:cb:d9:4b:1f:43:28:0b:19:12:f4:
5f:b8:53:49:93:b2:ef:37:61:0a:ec:d1:11:10:e2:
40:bc:1b:c3:74:e2:83:a8:24:32:0e:e8:0e:6f:5e:
f2:44:6a:27:40:4a:c0:f1:4c:98:ab:e3:52:18:2c:
fd:80:ff:23:9b:2f:77:8e:3e:20:a1:ee:df:82:24:
a1:f3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
DNS:izuf68thdbm0n4j5qywd7sz
Signature Algorithm: sha256WithRSAEncryption
a1:d6:4a:86:13:8e:36:a3:c2:ff:6a:e3:50:ce:48:97:19:a0:
d8:94:99:47:53:49:75:6f:27:15:ad:4a:b3:4c:50:5c:79:15:
d3:f7:55:26:f8:58:d2:77:26:c3:6c:8a:2d:46:58:df:5f:70:
40:54:4d:0a:7e:16:b2:b9:f6:6b:ce:ae:81:94:3f:88:b9:b3:
56:e5:1c:55:f1:97:7b:50:66:f3:19:c5:48:55:2d:22:60:6d:
36:0f:4b:99:ef:53:88:2d:3f:6a:47:2d:54:96:a9:35:2b:71:
7c:18:86:bc:a2:33:2a:b5:b5:ab:19:3b:85:f5:c8:2a:4c:9c:
54:71:ca:2b:14:00:a3:02:a3:6a:f8:fb:4f:40:d7:a2:59:18:
9c:7a:93:2b:8d:39:26:1c:42:b1:62:6e:55:dd:c5:48:fe:45:
cb:81:d1:bb:8a:86:05:80:9d:32:ec:da:cf:9c:83:fa:9b:f3:
90:70:38:56:c7:1d:7d:e6:69:91:e2:90:77:db:20:50:43:f6:
8d:5d:7f:52:e7:eb:fc:9d:8e:75:91:f6:63:b6:b9:96:2a:ef:
0f:1f:99:13:4a:d6:5d:72:d7:1a:a8:71:0f:b6:21:21:a6:81:
40:e2:74:f4:89:cd:0e:ae:24:0b:e2:c2:07:69:1c:06:0d:ad:
3b:3f:5e:a2
Can you just check that you've checked every client that appears when you run this query?
max(apiserver_request_count) by(client)The values don't matter it's just an aggregation so we can see all clients requesting against the API.
ne1000
commented
5 years ago @brancz
if the values don't matter, how to ignore the alert in my case?
brancz
commented
5 years ago You can always just silence an alert in Alertmanager :slightly_smiling_face: nonetheless we should figure out what's up here.
willtrking
commented
5 years ago Noticed this as well when setting up on EKS.
I BELIEVE that EKS manages the renewal of certificates by itself, so I've removed the rule on my end.
kevtaylor
commented
5 years ago @brancz Was there any update on this? - we get spammed a lot on our alerts with this false detail
brancz
commented
5 years ago You can configure the certificate expiry thresholds https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/c0b31ea63564966021f9e6010090acded475b192/config.libsonnet#L42-L43
If you want to ignore it entirely though you can also just remove the alert, or silence it :slightly_smiling_face: .
kevtaylor
commented
5 years ago Hi. Thanks for this answer but...
We are using the helm chart which has this baked in: https://github.com/coreos/prometheus-operator/blob/master/helm/exporter-kubernetes/templates/kubernetes.rules.yaml
How do we influence that?
And if I do want to programmatically silence that alert, is there a way of doing so in this chart - selective alerting ?
brancz
commented
5 years ago I'm not aware of tooling to declaratively silence alerts, I agree that would be neat to have.
@gianrubio maintains the helm charts, the coreos/red hat team just maintains the jsonnet part, as we don't use helm. I can't help you with helm things unfortunately.
kevtaylor
commented
5 years ago @gianrubio Do you have any helm updates to fix this?
joshbranham
commented
5 years ago Yeah we are also seeing this issue even though all our certificates appear up to date, and it specifically states the apiserver. I updated the alert on our side (we are using version 0.17 still) to use the histogram as updated in: https://github.com/coreos/prometheus-operator/blob/0bad93292506ace68e344c9a991af6ae76ae1a51/contrib/kube-prometheus/manifests/prometheus-rules.yaml#L752-L759
The strange part is the values come and go..
kevtaylor
commented
5 years ago @brancz Is this project still active? We don't seem to get any responses
leoncard
commented
5 years ago We are observing the same behaviour with our clusters.
Does anyone have any news on that?
We're finding it quite weird because every certificate seems to be up to date but the metrics show something different.
We tried restarting the apiserver docker container on the master node following this comment.
The alert stopped for a while but came back hours after.
joshbranham
commented
5 years ago So we figured out our issue (at least it is the only thing that makes sense). The alert was firing sporadically, and it was only when our single user that had certificate-based authentication was communicating with the API (Jenkins). Those certs did, in fact, expire in line with what the alert was saying, and since rotated them we have not had the alert fire.
shovelend
commented
5 years ago @joshphp our histogram is incrementing sporadically as well, we couldn't tie them to any client machine yet, but we noticed that for the clusters these numbers are increasing they are increasing at the same exact time. May I ask how you traced down that single user?
kevtaylor
commented
5 years ago I think also that this PR may be related to this issue: prometheus-operator/prometheus-operator#2058
joshbranham
commented
5 years ago @shoveland the old fashioned way: the user stopped being able to talk to the API with certificate warnings 😏
brancz
commented
5 years ago It seems that this issue is about expired client certs after all :slightly_smiling_face: . I'll keep this open for now as prometheus-operator/prometheus-operator#2058 is correct, this alert is about client certs, not serving certs, however, that won't change it from firing, so you will need to check up on your certs and check the apiserver's logs for which clients did these requests with expired certs.
shovelend
commented
5 years ago @brancz that's correct, let's close this after rephrasing the description of the alert. Having checked the apiserver's logs we still don't see the culprit, there are no messages logged regarding expired certificates nor authentication requests that failed. Do you have any other ideas where we could identify the client?
brancz
commented
5 years ago I realize this is drastic but if I recall correctly if you bump logging verbosity to --v=10 then you see the user identity of the certificate printed. That's the best I can do, otherwise I'd suggest we should add a log line to Kubernetes to log this more explicitly.
shovelend
commented
5 years ago Setting the log verbosity to 10 helped us track down the issue.
We found out that the client certificate data (part of the ./kube/config) we generated for developers was expiring after 1 month.
When developers were trying to access the kubernetes-dashboard by kubectl proxy-ing from their local machines (with an expired certificate), the metric increased as well. We extended the expiry date of the generated certificates.
This issue was quite difficult to track down and we found the needle in the haystack when setting the apiserver verbosity to 10 (had to go through 100.000 lines of log looking for a possible culprit.) If we were to suggest a place for improvement, it would be the alert message. If the metric could capture the url that was about to be accessed with the expired certificate and/or the client IP address, it would be incredibly helpful and could be displayed within the alert.
ChristofferNicklassonLenswayGroup
commented
5 years ago @shovelend could you share a logline that would help me find it in my logs :)
daimoniac
commented
5 years ago The metric could tell us which certificate is about to expire.
brancz
commented
5 years ago @daimoniac I don't think that's a good idea, that would allow clients to produce arbitrary amounts of metrics leading to a denial of service attack against a kubernetes API. I think we should add an info log line that says which user/certificate caused the counter to increment. Happy to review your Kubernetes PR if you open this :slightly_smiling_face: .
itaysk
commented
5 years ago I'm running prometheus-operator on EKS and have this issue as well. If I understand correctly (couldn't find anything on the web about this metric apiserver_client_certificate_expiration_seconds_bucket) this is about api clients (such as kubectl) using soon to be expired certs.
I don't think it's actually kubectl in this case since in EKS it's using a custom webhook validator.
Any pointers to how I can identify the aged certs, given I don't have access to the control plane?
If this is the by-design behavior of EKS (see @willtrking 's comment), then should this alert be preconfigured? consider the more general case outside EKS of automatically rotating short lived certs.
willejs
commented
5 years ago @itaysk I have just taken the system-alerts rules out as this is managed by AWS. I couldn't find a good way to exclude these specific rules, i am more intrigued as to why they are at 0, i suspect the prom library they use does this by default? It probably shouldnt.
itaysk
commented
5 years ago @willejs - didn't what to remove the entire kubernetes-system rules set (https://github.com/helm/charts/blob/master/stable/prometheus-operator/templates/alertmanager/rules/kubernetes-system.yaml) for now i've edited those rules out.
azalio
commented
5 years ago Setting the log verbosity to 10 helped us track down the issue.
We found out that the client certificate data (part of the
./kube/config) we generated for developers was expiring after 1 month. When developers were trying to access thekubernetes-dashboardbykubectl proxy-ing from their local machines (with an expired certificate), the metric increased as well. We extended the expiry date of the generated certificates.This issue was quite difficult to track down and we found the needle in the haystack when setting the apiserver verbosity to 10 (had to go through 100.000 lines of log looking for a possible culprit.) If we were to suggest a place for improvement, it would be the alert message. If the metric could capture the
urlthat was about to be accessed with the expired certificate and/or the client IP address, it would be incredibly helpful and could be displayed within the alert.
Can you explain which words you searched?
madAndroid
commented
5 years ago We're seeing the same issue on our clusters and finding it incredibly hard to track down the expiring certificate - can someone please share a log line we can search for when the verbosity is set to 10?
azalio
commented
5 years ago We're seeing the same issue on our clusters and finding it incredibly hard to track down the expiring certificate - can someone please share a log line we can search for when the verbosity is set to 10?
Just try to search word cert
stale[bot]
commented
5 years ago This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.
andrewsav-bt
commented
4 years ago @daimoniac I don't think that's a good idea, that would allow clients to produce arbitrary amounts of metrics leading to a denial of service attack against a kubernetes API.
So you are saying, that we are producing alert of a metric that can be caused by a couple a dozen clients (see screenshot from ne1000 above), we have no way to identify which client caused it, and providing this information will cause denial of service?
On the face value it does not sound right, could you please explain in more details what are different moving parts here are and what exactly makes it impossible to provide useful, exact information?
Failing that, how to identify the culprit client manually, without checking them all?
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.
AndrewSav
commented
4 years ago /remove stale
dlandtwing
commented
4 years ago Failing that, how to identify the culprit client manually, without checking them all?
Had the same issue on our openshift cluster today. I've found the responsible clients by capturing network traffic on the apiserver https port using tcpdump and analyzing the tcp handshakes using wireshark (filter "tls.handshake.client_cert_vrfy.sig") and inspecting the client certificates
It turned out to be expiring kubelet client certificates (cn system:nodes...) causing these issues.
by running
openssl x509 -in /etc/origin/node/certificates/kubelet-client-current.pem -noout -startdate -enddate
on each node we were able to identify which kubelet certificates were about to expire
Ultimately the solution was obtaining all pending CSRs and approving them manually:
oc get csr --sort-by='{.metadata.creationTimestamp}'
oc adm certificate approve csr-thgqd
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.
xavierW
commented
3 years ago Failing that, how to identify the culprit client manually, without checking them all?
Had the same issue on our openshift cluster today. I've found the responsible clients by capturing network traffic on the apiserver https port using tcpdump and analyzing the tcp handshakes using wireshark (filter "tls.handshake.client_cert_vrfy.sig") and inspecting the client certificates
It turned out to be expiring kubelet client certificates (cn system:nodes...) causing these issues. by running
openssl x509 -in /etc/origin/node/certificates/kubelet-client-current.pem -noout -startdate -enddateon each node we were able to identify which kubelet certificates were about to expireUltimately the solution was obtaining all pending CSRs and approving them manually:
oc get csr --sort-by='{.metadata.creationTimestamp}'oc adm certificate approve csr-thgqd
is there any easier way to find out the expired client ?
coderanger
commented
3 years ago Failing that, how to identify the culprit client manually, without checking them all?
Had the same issue on our openshift cluster today. I've found the responsible clients by capturing network traffic on the apiserver https port using tcpdump and analyzing the tcp handshakes using wireshark (filter "tls.handshake.client_cert_vrfy.sig") and inspecting the client certificates It turned out to be expiring kubelet client certificates (cn system:nodes...) causing these issues. by running
openssl x509 -in /etc/origin/node/certificates/kubelet-client-current.pem -noout -startdate -enddateon each node we were able to identify which kubelet certificates were about to expire Ultimately the solution was obtaining all pending CSRs and approving them manually:oc get csr --sort-by='{.metadata.creationTimestamp}'oc adm certificate approve csr-thgqdis there any easier way to find out the expired client ?
Not via metrics. Metrics are for numeric values, not textual data like client names. As mentioned earlier in the thread, apiserver should be logging more verbosely when an almost-expired client connects but that fix would need to be in apiserver itself, not here.
bpinske
commented
3 years ago This is a frustrating problem. Its unclear to me how the creation of this histogram could be fixed without introducing cardinality problems. My first thought was to add a label of the hex representation of the source IP. Then a few other ideas that technically satisfy the requirement that all prometheus metrics be integers, but nothing that would ever be accepted upstream. https://github.com/kubernetes/kubernetes/commit/49a19c6011e05363a8baf8e99c917d11a9496568
pcap dumping is one way to figure this out as noted above, possibly the best. But remember you may have many API servers behind an LB so may want to dump all of them.
The way we've just found our mystery cert is with the following splunk query. You may have to adapt this slightly if you use elk or something but the idea should be the same: search audit records for usernames. The built-in cert rotation mechanism of normal k8s components like the kubelets should be auto fixing anything with the system: prefix so exclude those. In my case I got lucky that this only showed a small number of username possibilities. We use OIDC for human kubectl auth, rather than x509 certs, so the logs were sparse for me.
index="main" apiVersion="audit.k8s.io/v1" * | spath "user.username" | search "user.username" != "system:*"
github-actions[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.
drasikhov
commented
1 year ago Hi! I'm new to Kubernetes and I'm sorry if it's a dumb question, but how exaclty the apiserver_client_certificate_expiration_seconds histogram and metric can be useful? We can't really tell, who was that client with the expiring certificate, so what's the big point in it then?
It would be much better if we would have metrics (like at least notAfter property) for each certificate that is used in the cluster directly in the API (for all components, like etcd/front-proxy/controller manager etc.) the same way we can get them with kubeadm certs check-expiration. If you have another etcd or some other component on some node, you should be able to retrieve that data through API as well. And metric's description should clearly state what it's puprose.
I understand that with kubeadm you get these on the file level from PKI directory, but this just looks very strange that you can't really get this data natively through API and that in such cases you have to use 3rd party solutions, like x509-certificate-exporter.
ykfq
commented
1 year ago I managed to fix this by upgrade prometheus from 2.37.1 to current latest version 2.44.0（only tested on this version）
aostrovsky
commented
1 year ago I managed to fix this by upgrade prometheus from 2.37.1 to current latest version 2.44.0（only tested on this version）
running 2.45.0 in aks 1.24.10 Still getting those alerts
h0jeZvgoxFepBQ2C
commented
1 year ago Still receiving these alerts
jvstein
commented
10 months ago I ran into a weird setup where I had both a token as well as a client-certificate-data and client-key-data section in a ~/.kube/config file on a client.
From the user perspective everything was working fine, but it must have been attempting requests with the certificate first and then silently falling back to the working token value, not showing any error to the user.
This client was constantly triggering the KubeClientCertificateExpiration alert whenever it was active. The histogram_quantile portion of the alert query (below) was showing a straight line at 0.
histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[2m]))) < 86400Basic structure of the bad client config.
users:
- name: the-user
user:
token: <VALID_TOKEN>
client-certificate-data: <EXPIRED_CERT>
client-key-data: <KEY>This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.
github-actions[bot]
commented
4 months ago This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.
What did you do? wget https://codeload.github.com/coreos/prometheus-operator/tar.gz/v0.23.1 install prometheus-operator use kubectl create -f prometheus-operator-0.23.1/contrib/kube-prometheus/manifests/ || true
What did you expect to see? all components work correctly.
Environment
K8s version: v1.11.0
Prometheus Operator version: v0.23.1
Kubernetes cluster kind:
Manifests:
I used cfssl generate pem and keys
my k8s cluster and prometheus seems fine. but KubeClientCertificateExpiration always trigger alert, how do I fix it ?