search

headlamp-k8s / headlamp

A Kubernetes web UI that is fully-featured, user-friendly and extensible
https://headlamp.dev
Apache License 2.0
2.26k stars 159 forks source link

Pod metrics button in server mode deployment not working #1661

Open dioniseo opened 9 months ago

dioniseo commented 9 months ago

Hello, it seems like the new feature of showing high level pod metrics not properly working in case of restricted rights of the user. My user has cluster view and some specific namespaces admin rights (but I don't have rights for monitoring namespaces where the Prometheus is deployed). This causes issues like following when I try to click on this button:

{ "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "pods \"prometheus-prometheus-kube-prometheus-prometheus-0\" is forbidden: User \"sso:my-user-name\" cannot get resource \"pods/proxy\" in API group \"\" in the namespace \"monitoring\"", "reason": "Forbidden", "details": { "name": "prometheus-prometheus-kube-prometheus-prometheus-0", "kind": "pods" }, "code": 403 }

This could be fixed with allowing additional RBAC permissions for the user, but that also could bring some security concerns as this way via proxy user will be able to access all the pods in the namespace where Prometheus was deployed and also potentially could connect to these pods bypassing authentication and getting application metrics and also that will require additional RBAC changes, so in general we'd like to avoid this.

Maybe better and simpler approach could be to use the Kubernetes Metrics Server as source of these basic monitoring data in case of in cluster deployment (I believe its already used for getting node metrics - memory/cpu utilization on the main dashboard)? It seems it could provide all this required info and it should not be required to have additional rights for scraping metrics from this (as node metrics are showing just fine). Also Prometheus may have different deployment approaches and could include multiple tools for authorization and authentication and sometime it could block requests from Headlamp without proper authentication in it so from this point of view, so it seems like Metrics API could be simpler solution for getting these base metrics for showing on the Pods summary page?

Could you please review this possibility for enhancement?

Headlamp version: 0.22.0

Thanks!

joaquimrocha commented 9 months ago

@yolossn , any thoughts on this?

dioniseo commented 9 months ago

@joaquimrocha also it seems like because of adding this new feature, we started to see the out of memory error on the page after its stay active for ~10-15 mins:

image

This issue didn't occur in previous version of Headlamp, so that could be related to the Prometheus plugin that was added. We had to rollback to the previous version due to this issue.

joaquimrocha commented 9 months ago

@dioniseo , sorry to hear that. We'll definitely look into that issue next week. Would it be helpful if there was a way to disable certain plugins using a configuration file? To apply in cases like this till there's a fix.

dioniseo commented 9 months ago

@joaquimrocha yes I think that would be helpful, are there any wiki/instructions on how we could enable/disable specific plugins in already built image?