Closed miminar closed 2 months ago
Thanks @miminar. I agree. After reflecting on this, I think it would be best to go with this PR since I have already set the expectations for this metric. For ephemeral_storage_container_{hard,soft}_limit_percentage
I like this for V2 but I don't want to break default/expect behavior before then.
I do have slight reservations around high cardinality risk, but we do have relabeling the user can use and at worst, I can add an option flag to disable the source
label by default.
Great job! If you want to contribute your PrometheusRule
after this I am open to PRs.
Thanks!
@miminar This is now release in 1.11.0
Thank you! It helps a lot.
in order to differentiate between "hard" and "soft" limits
"hard" comes
pod.spec.containers.resources.limits["ephemeral-storage"]
(if set) and terminates the pod once reached"soft" comes from the node and may not necessarily cause an eviction of the pod in question
Motivation
To implement an alert rule à la KubePersistentVolumeFillingUp for a container like this:
Notice the middle part of the expression (
and on (pod_namespace, pod_name, exported_container) ( max_over_time( ... > 0.06)
which makes the whole evaluation fragile, complicated and slow.It is needed to filter out all containers with
pod.spec.containers.requests.limits["ephemeral-storage"]
unset.Without it, there would be too many instances of the alert generated for a single node running out of ephemeral storage (one for each container running on the node). For that case, there should be another alert like
NodeOutOfEphemeralStorage
With this change in place, one could simplify the expression to:
I don't have a strong opinion on the new label name/ values. Instead of adding a new label, there could be a whole new metric instead (e.g.
ephemeral_storage_container_hard_limit_percentage
).Please let me know what you think.