search

kubernetes / dashboard

General-purpose web UI for Kubernetes clusters
Apache License 2.0
14.27k stars 4.14k forks source link

Kubernetes Dashboard does not show CPU Usage and Memory Usage graph if namespace has Cron Jobs #4145

Closed tchellomello closed 4 years ago

tchellomello commented 5 years ago
Environment

I've deployed the kubernetes dashboard as documented at https://github.com/kubernetes/dashboard/releases/tag/v2.0.0-beta2 and got it working as expected.

The metrics-scaper is working as expected and I can see the graph displayed below:

image

However, if a namespace has a Cron Job defined, the top 2 graphs (CPU and Memory Usage) are not displayed due to the error below:

[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 Getting list of all cron jobs in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Incoming HTTP/2.0 GET /api/v1/daemonset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/deployment/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all deployments in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/job/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all jobs in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/pod/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all pods in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/replicaset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all replica sets in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/replicationcontroller/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all replication controllers in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/statefulset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all pet sets in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/ingress/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/service/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all services in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/configmap/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/persistentvolumeclaim/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list config maps in the namespace kubeapps 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list persistent volumes claims 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/secret/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of secrets in &{[kubeapps]} namespace 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting pod metrics 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Skipping metrics because of error: Received invalid number of resources from sidecar. Expected 10 received 0 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code

Then the dashboard is displayed like this:

image

Installation method: kubectl apply
Kubernetes version: v1.14.4
Dashboard version: v2.0.0-beta2+0.ge9d82f7a
Operating system: CentOS 7
Node.js version ('node --version' output):
Go version ('go version' output):
Steps to reproduce
  1. Deploy the dashboard 
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta2/aio/deploy/recommended.yaml
  2. Deploy a test pod and the graph will be displayed
  3. Add a cronjob to the namespace and the graph will not be displayed.
Observed result

CPU and Memory Usage graphs are not displayed when a cronjob is listed

Expected result

CPU and Memory Usage graphs should be displayed when a cronjob is listed

Comments
tchellomello commented 5 years ago

Just for the documentation purposes, it is still happening on v2.0.0-beta3+0.g677aff93

maciaszczykm commented 5 years ago

@tchellomello We are aware, we have a couple of issues with metrics that are not fixed yet. We will work on them.

davidtseng commented 5 years ago

I also see the CPU/memory graphs not show up if there is a crashlooping pod in the namespace.

maciaszczykm commented 5 years ago

The issue is not directly caused by Cron Jobs. [kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported does not appear anymore.

There is an issue with pods created by cron jobs and with crashlooping pods.

maciaszczykm commented 5 years ago

@jeefy I think we have a problem with metrics scraper here. When the pod is completed (i.e. job's child) or in a crashed state, we are dealing with 2019/09/05 15:22:04 Skipping metrics because of error: Received invalid number of resources from the sidecar. Expected 3 received 2. Could you take a look on this as you are more familiar with the code both on the dashboard and scraper side?

jeefy commented 5 years ago

I'll take a look at this soon. Thanks!

clhlc commented 4 years ago

+1

lefranco6910 commented 4 years ago

I just got the same issue with 2.0.0beta4 today. I found this thread by googling the issue. I thought it might be good to share. Thanks for looking at this issue.

maciaszczykm commented 4 years ago

Any update, @jeefy?

SMelnykov commented 4 years ago

Dashboard v2.0.0-beta4+0.gf232c589 - Not see any graphs on Cluster -> Nodes page. See graphs on the name space (All namespaces) -> Overview page. kubectl top node --> See performance information per nodes Will be good see memory in percentage also not in Mi

strainovic commented 4 years ago

+1 same issue with beta5

floreks commented 4 years ago

We are aware of that. That's why it is not closed.

teeram commented 4 years ago

Thanks for working on this guys. I'm having this problem too.

sompholyai commented 4 years ago

Do you have solution ?

kaikuchn commented 4 years ago

The bug has been fixed with https://github.com/kubernetes/dashboard/pull/4506 Using 2.0.0-rc2 works. If you still don't get metrics maybe your issue is something else.

ysyyork commented 3 years ago

i was not able to login anymore after I changed from 2.0.0 to 2.0.0-rc2 and when I was checking the metric scraper pod i got this. 

{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2021-04-18T09:42:42Z"}

And I got stuck on the token page. After I submitted token the dashboard pod returned 200 but it's not redirecting me to the dashboard.

This seems related to issue #3322 but I also tried their solution including using https, nothing worked. When I change back to v2.0.0, i can login but the above error message is still there. Anyone can give a bit hint on what does that message mean?

robozb commented 2 years ago

I experienced the same problem with these as well:

root@kubernetes-master-ubuntu:~# k version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

Dashboard UI screenshot: https://drive.google.com/file/d/1-BPEynBtVV8pq42jZnXjpOOhC2LFgqER/view?usp=sharing

Thank you so much for your work!

Bye: Beci

floreks commented 2 years ago

You need to have metrics-server installed and working correctly.

robozb commented 2 years ago

You need to have metrics-server installed and working correctly.

OK, thank you so much I'm checking!