elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.66k stars 8.23k forks source link

[Stack Monitoring] Disk usage threshould alert should support absolute byte values too #123964

Open a03nikki opened 2 years ago

a03nikki commented 2 years ago

Describe the feature: Match Stack Monitoring's "disk usage threadshold" alert configuration options with the Elasticsearch configuration options.

Describe a specific use case for the feature: If Elasticsearch nodes have very large disks, the percentage based alerts are not optimal. Often users use absolute values for their wartermarks. However, if they wanted to set-up Alerts to mimic the settings with the out of the box (OOTB) content, they cannot.

image (Screenshot is from 7.16.3 cluster on cloud.elastic.co)

From https://www.elastic.co/guide/en/kibana/master/kibana-alerts.html#kibana-alerts-disk-usage-threshold

Disk usage threshold

This rule checks for Elasticsearch nodes that are nearly at disk capacity. By default, the condition is set at 80% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day.

From https://www.elastic.co/guide/en/elasticsearch/reference/7.16/modules-cluster.html#disk-based-shard-allocation

... cluster.routing.allocation.disk.watermark.low

(Dynamic) Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated. ...

elasticmachine commented 2 years ago

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)