search

rptaylor / kapel

APEL accounting for Kubernetes.
GNU General Public License v3.0
3 stars 3 forks source link

simplified pod resource metrics (support for initContainers) #71

Open rptaylor opened 1 week ago

rptaylor commented 1 week ago

Currently I believe initContainers will not be accounted or seen, since there are separate metrics like kube_pod_init_container_resource_requests for them: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md

It would be a hassle to write code to add up the initContainers, along with the regular containers, and do a max on them to try to duplicate/emulate the logic that k8s follows to determine the final pod resource amounts. It would be much better to simply query the actual resource amount of the whole pod, which is also recommended under the kube_pod_container_resource_requests description: "It is recommended to use the kube_pod_resource_requests metric exposed by kube-scheduler instead, as it is more precise."

Info about the kube-scheduler metrics: https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/#kube-scheduler-metrics I checked on one of our clusters and kube_pod_resource_requests was not available. I think those metrics may need to be enabled with the --show-hidden-metrics-for-version flag, or maybe Prom needs to be configured to scrape scheduler metrics. That would be an extra complication for deployment, but it's probably worth it to keep the code simpler and less bug prone, especially if support for accounting initContainers is needed.

rptaylor commented 1 week ago

That being said https://kubernetes.io/docs/reference/instrumentation/metrics/ says that kube_pod_resource_request is a STABLE metric (even going back to v1.27) ...

Update: but this says, referring to kube_pod_resource_request,

The metrics are exposed at the HTTP endpoint /metrics/resources and require the same authorization as the /metrics endpoint on the scheduler. You must use the --show-hidden-metrics-for-version=1.20 flag to expose these alpha stability metrics.
mwestphall commented 6 days ago

@rptaylor I've also confirmed that kube_pod_resource_request isn't present on CHTC's v1.27 k8s cluster. To attempt to approximate that, do you think it would make sense to take the max of the union of the sum of the init_container and container resource requests?

max by (pod) (
    sum by (pod) (kube_pod_container_resource_requests{namespace="<ns>",resource="cpu"}) 
    OR 
    sum by (pod) (kube_pod_init_container_resource_requests{namespace="<ns>",resource="cpu"})
)
rptaylor commented 5 days ago

Maybe, but I'd like to at least investigate the alternative first (kube-scheduler metrics) to see how feasible it is, especially if the KSM developers specifically recommend that way and it would make our lives easier too. I suspect the metrics are there but we just need to do something to see them and get them in Prometheus.

https://yuki-nakamura.com/2023/10/21/get-kube-schedulers-metrics-manually/