kubernetes-monitoring / kubernetes-mixin

A set of Grafana dashboards and Prometheus alerts for Kubernetes.
Apache License 2.0
2.11k stars 598 forks source link

Incomplete usage of metric kube_resourcequota #662

Closed piotrminkina closed 2 weeks ago

piotrminkina commented 3 years ago

Hello

I'm using release-0.8 of kubernetes-mixin, kube-state-metrics v2.1.1 and kubernetes v1.20.8. I think usage of metric kube_resourcequota is incomplete in the following part of code:

https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/7120319a5b5c45e8dd2e79f0ad60e2284c6d6f1b/dashboards/resources/namespace.libsonnet#L83-L86

I think it should be something like this:

      local cpuQuotaRequestsQuery = 'scalar(kube_resourcequota{%(clusterLabel)s="$cluster", namespace="$namespace", type="hard", resource=~"requests.cpu|cpu"})' % $._config;
      local cpuQuotaLimitsQuery = 'scalar(kube_resourcequota{%(clusterLabel)s="$cluster", namespace="$namespace", type="hard", resource="limits.cpu"})' % $._config;
      local memoryQuotaRequestsQuery = 'scalar(kube_resourcequota{%(clusterLabel)s="$cluster", namespace="$namespace", type="hard", resource=~"requests.memory|memory"})' % $._config;
      local memoryQuotaLimitsQuery = 'scalar(kube_resourcequota{%(clusterLabel)s="$cluster", namespace="$namespace", type="hard", resource="limits.memory"})' % $._config;

As described in https://kubernetes.io/docs/concepts/policy/resource-quotas/#compute-resource-quota, we may use "cpu" or "requests.cpu" and "memory" or "requests.memory", but queries from kubernetes-mixins omits "cpu" and "memory".

For tests I applied the following resource:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: rq
spec:
  hard:
    cpu: "0.5"
    memory: 512Mi
    requests.cpu: "0.75"
    requests.memory: 768Mi
    limits.cpu: "1"
    limits.memory: 1Gi

Next I got status of this resource:

# ...
status:
  hard:
    cpu: 500m
    limits.cpu: "1"
    limits.memory: 1Gi
    memory: 512Mi
    requests.cpu: 750m
    requests.memory: 768Mi
  used:
    cpu: 200m
    limits.cpu: "4"
    limits.memory: 2Gi
    memory: 256Mi
    requests.cpu: 200m
    requests.memory: 256Mi

As you can see, used cpu and requests.cpu are the same, similar to used memory and requests.memory. Strange situation is with hard cpu and requests.cpu — in this case I don't see what value is priority for k8s. Same situation is with hard memory and requests.memory.

I reported this problem because in my organisation we are using hard cpu and memory. Without proposed changes we can't see requests quotas on Grafana Dashboards.

Regards Piotr Minkina

github-actions[bot] commented 2 weeks ago

This issue has not had any activity in the past 30 days, so the stale label has been added to it.

Thank you for your contributions!

skl commented 2 weeks ago

Fixed by #962