When any of the log* nodes report excessive disk usage and a sudo du -kx --max-depth=3 / | sort -rn | head -10 shows the culprit is under /var/lib/elasticsearch, then the typical solution is to manually prune off the oldest logstash indices (created by day) to alleviate disk space.
On the node being fixed, first obtain the list of indices in basically date sorted order:
curl http://localhost:9200/_cat/indices|sort -k3
Looking at the third column, start working down the list of "logstash-*" indices performing an index delete operation until enough space is recovered (using df)
When any of the
log*nodes report excessive disk usage and asudo du -kx --max-depth=3 / | sort -rn | head -10shows the culprit is under/var/lib/elasticsearch, then the typical solution is to manually prune off the oldest logstash indices (created by day) to alleviate disk space.On the node being fixed, first obtain the list of indices in basically date sorted order:
Looking at the third column, start working down the list of "logstash-*" indices performing an index delete operation until enough space is recovered (using
df)An example of a deletion is