Open sidewinder12s opened 2 months ago
As far as I can tell that seems to (still) be a valid value for a PodCondition. You can find (a little bit) more about it in Pod conditions.
I was suspecting that, just kind of surprised I hadn't seen that status actually in any of our metrics as we get containers in that state fairly regularly.
It's unfortunate KSM exposes these like this, the cardinality caused by it is extreme.
It's unfortunate KSM exposes these like this, the cardinality caused by it is extreme.
On the one hand, I don't think it would be wise to omit such label value given that it seems to still be a valid status. On the other hand, I can see how this might not be of interest and could impact cardinality. I wonder if we could have some kind of metric cardinality enforcement mechanism like we have for kube-apiserver, though I'm not sure if it would be too much just for this one metric/label. There are probably better ways to solve this, though I can't think of anything right now.
I was suspecting that, just kind of surprised I hadn't seen that status actually in any of our metrics as we get containers in that state fairly regularly.
You mean you often have containers with unknown
Ready condition status, yet they don't show up as such in your metrics?
KSM has a lot of the status metrics that are similar to this one where it has many label + value combinations for each pod that can explode cardinality and cost, especially if you are using a metrics vendor which bills by the cardinality/active series. So I think some broader cardinality enforcement might be broadly useful.
As far as Ready Status, I might be mixing ContainerStatusUnknown
that I see on pods when I run kubectl get pods
for example, with Ready status which indeed might be different.
The metrics that have conditions are some are our most problematic ones as well. I've been thinking of two ways to address them:
kube_pod_status_ready
, return a value -1=Unknown, 0=False, 1=True and remove the "condition" label. True
combination and the condition is actually True. This would help us for metrics like kube_node_status_condition
and kube_pod_status_phase
where we have many nodes conditions/pods phases, but vast majority are always 0 and we only really care about conditions that are True.For the True/False/Unknown cases like
kube_pod_status_ready
, return a value -1=Unknown, 0=False, 1=True and remove the "condition" label.
Some of these metrics are marked as stable, so removing a label might not be something we would want to do. However, I can't tell if kube-state-metrics really follows the Kubernetes metrics stability framework.
Ah. Another option might be:
For 2, one of our metrics vendors supports dropping data based on metric value, which we've implemented for many of our high cardinality metrics that are mostly zeros.
So far, many of them tend to fall into 2 camps:
When we were adding these drops, the main drawback was that any gauges/math might not work without the zero value, but can be worked around.
For 3, I had wondered if anyone had proposed/questioned how overloaded some of these status metrics become with the # of labels/label combinations that end up being created. I don't really think there'd be huge downside to breaking these up into individual metrics vs the current solution.
Would another option be to have a runtime flag to just not emit the metrics if they are zero?
Dropping zeros would be very useful. If there was an option to enable for KSM, that would cut out about ~55% of the series that we don't use.
I don't really think there'd be huge downside to breaking these up into individual metrics vs the current solution.
Agree. A metric like kube_pod_status_ready
has the same information encoded twice. One series with condition=true and value 1, is the same as condition=false and value 0.
kube_pod_status_phase
could be broken out to kube_pod_status_phase_pending
, kube_pod_status_phase_succeeded
, kube_pod_status_phase_failed
and kube_pod_status_phase_running
. With ability to drop 0 values, that would also cut out a lot excess data. This structure would also work better with the denylist
to be more fine grained in what metrics are kept.
I do wonder if there is some use case to this structure we might be missing. But at scale the way these metrics are laid out is egregious.
/triage accepted /kind support
/remove-kind bug
/assign @dgrisonnet
What happened:
What is the purpose of the unknown condition on
kube_pod_status_ready
metric?Searching through my metrics for a fairly high number of pods and nodes, I don't see that status ever being true.
So my initial impression is this is needlessly wasting cardinality in a way that is very hard to avoid.
Anything else we need to know?:
Environment:
kubectl version
): 1.28