Lens autodetects my Prometheus installation but doesn't display metrics

[Starttoaster]

Describe the bug Lens autodetects my Prometheus installation but doesn't display metrics. I have kube-state-metrics installed in the cluster and node-exporter running on the hosts. The prometheus instance that Lens detects is scraping those exporters, but Lens isn't able to find them. The Lens documentation doesn't list anything as far as metric relabelings it requires or really any configuration details that may be required to get it to work.

To Reproduce Steps to reproduce the behavior:

1. Have a custom kube-prometheus installation
2. Have a kube-state-metrics installation
3. Have node-exporter installations on all of your kubernetes nodes
4. Have Lens autodetect your Prometheus instance, confirm in cluster settings in the Metrics menu it has autodetected the prometheus-operated service to your running instance.
5. See no metrics in Lens.
6. See no documentation about how to configure your exporters, your kube RBAC permissions, or Prometheus for Lens to query Prometheus for those metrics.
7. Do some quick Googling and find several Issues related to users experiencing Issues getting Prometheus metrics to work with their Lens installations but nothing exactly relevant to me that I found.
8. Make an Issue on the Lens repository.

Expected behavior I expect to be able to see my node/Pod metrics contained by Prometheus.

Screenshots I can add some if needed but not sure which pictures would be useful to add onto my description.

Environment (please complete the following information):

• Lens Version: 2024.5.271333-latest
• OS: [e.g. OSX] MacOS Sonoma 14.5
• Installation method (e.g. snap or AppImage in Linux): DMG

Additional context This is an on-prem cluster if that matters. My local user has RBAC permissions to port forward Services. Perhaps not proxy those services though.

[Starttoaster]

Oh interesting, bit of extract context in the logs if opening up from the CLI.

warn:    ┏ [METRICS-ROUTE]: failed to get metrics for clusterId=redacted: Metrics not available +37ms
warn:    ┃ [  1] Error: Metrics not available
warn:    ┃ [  2]     at loadMetricHelper (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11886:19)
warn:    ┃ [  3]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
warn:    ┃ [  4]     at async Promise.all (index 9)
warn:    ┃ [  5]     at async /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11929:28
warn:    ┃ [  6]     at async Object.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3584:31)
warn:    ┃ [  7]     at async Router.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3418:5)
warn:    ┃ [  8]     at async LensProxy.handleRequest (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3385:7)
warn:    ┃ [  9] Cause:
warn:    ┃ [ 10] 'Error: Failed to POST /api/v1/namespaces/prometheus/services/prometheus-operated:9090/proxy/api/v1/query_range for clusterId=redacted: Forbidden\n' +
warn:    ┃ [ 11]   '    at /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:922:15\n' +
warn:    ┃ [ 12]   '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
warn:    ┃ [ 13]   '    at async loadMetricHelper (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11880:18)\n' +
warn:    ┃ [ 14]   '    at async Promise.all (index 9)\n' +
warn:    ┃ [ 15]   '    at async /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11929:28\n' +
warn:    ┃ [ 16]   '    at async Object.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3584:31)\n' +
warn:    ┃ [ 17]   '    at async Router.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3418:5)\n' +
warn:    ┃ [ 18]   '    at async LensProxy.handleRequest (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3385:7)'

[Tixon123]

Hello Starttoaster,

Thank you for reaching out to Lens support!

Thank you for reporting a bug.

We are working on your issue. Stand by for further updates.

Best Regards, Tikhon Kudinov Lens Support Engineer Mirantis, Inc

[Starttoaster]

I was able to get the CPU, memory, network, and filesystem usage graphs to show up for Pods by adding the following to my ClusterRole that I assume when I authenticate to my clusters:

  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["prometheus-operated:9090"]
    verbs: ["get", "list", "watch", "create"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list"]

Though some problems still persist. As you can see in the following screenshot, the graphs show up, but in the Pod list the CPU and Memory columns are still blank:

Additionally, the Nodes list view doesn't display node level metrics despite all of them having node_exporter installed, and all of their node_exporters are being scraped by the Prometheus server in question:

When I run Lens from the CLI I get these logs now:

warn:    ┏ [METRICS-ROUTE]: failed to get metrics for clusterId=28bbac2f0cf17cfe3529a3c53426dcd7: Metrics not available +59ms
warn:    ┃ [  1] Error: Metrics not available
warn:    ┃ [  2]     at loadMetricHelper (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11886:19)
warn:    ┃ [  3]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
warn:    ┃ [  4]     at async Promise.all (index 15)
warn:    ┃ [  5]     at async /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11929:28
warn:    ┃ [  6]     at async Object.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3584:31)
warn:    ┃ [  7]     at async Router.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3418:5)
warn:    ┃ [  8]     at async LensProxy.handleRequest (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3385:7)
warn:    ┃ [  9] Cause:
warn:    ┃ [ 10] 'Error: Failed to POST /api/v1/namespaces/prometheus/services/prometheus-operated:9090/proxy/api/v1/query_range for clusterId=28bbac2f0cf17cfe3529a3c53426dcd7: Unprocessable Entity\n' +
warn:    ┃ [ 11]   '    at /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:922:15\n' +
warn:    ┃ [ 12]   '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
warn:    ┃ [ 13]   '    at async loadMetricHelper (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11880:18)\n' +
warn:    ┃ [ 14]   '    at async Promise.all (index 15)\n' +
warn:    ┃ [ 15]   '    at async /Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:11929:28\n' +
warn:    ┃ [ 16]   '    at async Object.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3584:31)\n' +
warn:    ┃ [ 17]   '    at async Router.route (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3418:5)\n' +
warn:    ┃ [ 18]   '    at async LensProxy.handleRequest (/Applications/Lens.app/Contents/Resources/app.asar/node_modules/@lensapp/core-main/dist/index.js:3385:7)'

Frankly, considering how finicky this bit of the Lens UI has been for me over the past year or two of using Lens, I'm a bit surprised Mirantis doesn't have documentation detailing the requirements to get the metrics to show up with a self-managed Prometheus instance and exporters. The magic combination of rituals I need to complete should probably be worth documenting for people.

Please let me know if you need any more information/logs.

[dark-brains]

Same issue here I have all components installed in the Cluster but cant see CPU usage metric. When using kubectl top node it shows CPU usage , I use kube-prometheus-stack helm chart all metrics available in dashboards but CPU usage for Nodes and Pods isn't available for Lens.

[igolka97]

Same issue using prometheus-operator. Also did this steps: https://github.com/lensapp/lens/blob/master/troubleshooting/custom-prometheus.md