Bump persistent disk alerts to closer to default, default is 90. The logsearch deployments will allow the data nodes to scale to 85% before preventing new shards from being placed. One of the nodes is at 85% as reported by curl -s 'localhost:9200/_cat/allocation?v' but prometheus sees the disk at 81%. So setting the prometheus alert threshold to 85% will silence the perfectly healthy logsearch data nodes with a couple percent to spare.
Bumping the high cpu warnings (seen mostly on diego cells) from the default of 10m to 30m since there basically isn't anything we can do or would do as a result of this warning.
Left the ephemeral disk alerts alone, the only one I found was from when I really did fill a disk on an emptvm, so these aren't noisy
Security considerations
Trying to reduce the noise so real issues are better seen.
Changes proposed in this pull request:
curl -s 'localhost:9200/_cat/allocation?v'but prometheus sees the disk at 81%. So setting the prometheus alert threshold to 85% will silence the perfectly healthy logsearch data nodes with a couple percent to spare.emptvm, so these aren't noisySecurity considerations
Trying to reduce the noise so real issues are better seen.