scholzj
commented
1 week ago I think this was repeatedly discussed in the past and rejected. I do not think we should issue any such warnings as there are many situations where best effort resources migth be desired.
Related problem
Kubernetes relies on a quality of service (QoS) classification based on resources configuration to make decisions about which Pods to evict when a Node is under pressure. The QoS classes are Guaranteed, Burstable, and BestEffort. When a Node runs out of resources, Kubernetes will first evict BestEffort Pods running on that Node, followed by Burstable and finally Guaranteed Pods. Only Pods exceeding resource requests are candidates for eviction.
Even if we have very good documentation about CPU and memory resources configuration, there are a number of users that do not configure them for Kafka and NodePools CRs in production deployments. There may be legitimate use cases for this, but most of the times it's simply an oversight which may cause Kafka cluster instability and, ultimately, a bad user experience.
Kubernetes Nodes have local ephemeral storage, usually backed by locally-attached writeable devices. Ephemeral means that there is no long-term guarantee about durability. Pods use ephemeral storage for scratch space, caching, and for logs, but it is not suitable for storing user data in production.
Kafka is a stateful application which relies on durable storage. It also provides data replication and tiered storage features, which help with rebuilding missing data due to Kafka node crash or restart, but there is no way to enforce proper replication. For that reason, the use of ephemeral storage brings higher risk of service or data loss, which may be acceptable in some temporary dev or test environment, but certainly not in production.
Suggested solution
We cannot enforce any minimum value for node's CPU and memory, also considering that there are other methods to do it (ResourceQuota/LimitRange), but it would be good to raise a log and status warning in case CPU or memory requests are not set at all. This means that we would raise a warning when Pod's qosClass is BestEffort, but not when it is Burstable or Guaranteed. Burstable makes sense to allow using the excess of CPU at the cost of a slightly increased eviction risk. It would also be good to raise a warning when ephemeral storage is used, highlighting the risk of service or data loss.
This resources configuration check would be enabled by default, with the possibility to opt out by setting a boolean environment variable. The new behavior would give users better chances to identify these misconfiguration early, maybe through some UI notification, before they actually cause a serious issue.
Alternatives
No response
Additional context
Some examples of issues reported by users: