search

sensu-plugins / sensu-plugins-elasticsearch

This plugin provides native ElasticSearch instrumentation for monitoring and metrics collection, including: service health and metrics for cluster, node, and more.
http://sensu-plugins.io
MIT License
32 stars 77 forks source link

Sensu checks cause high load to Elasticsearch 7 #163

Closed gerasym closed 5 years ago

gerasym commented 5 years ago

After upgrade our ES cluster from 6th to 7th version we noticed that 7th version can not handle the same query rate that 6th version was handling perfectly fine. The only thing that changed is ES version. Hardware, deployment, ingest rate and everything else did not change.

After investigation we have found out that the load is created by Sensu clients. We analyzed the queries and out from ~400 unique we did not find any that cause the load alone. Also we tried to replay the same queries with the same rate using python and elasticsearch library, and this way 7th version was handling queries as fine as 6th version did.

So it leads us to conclusion that the problem is caused not by what Sensu clients are sending but how they do it. To check this assumption we dumped the traffic from Sensu to ES and did not find anything suspicious. Additional problem could be that we have a lot of different Sensu clients versions running with different ES plugin versions :) And this can not be easily fixed.

Any thoughts or ideas about the possible causes will be highly appreciated. Thank you in advance.

gerasym commented 5 years ago

In the end we found out that load is caused by document security ES feature. Once it's configuration is removed - 7th version works fine. Sorry for wrong droids :)