search

haraka / haraka-plugin-elasticsearch

Ship Haraka log info directly to Elasticsearch
https://www.npmjs.com/package/haraka-plugin-elasticsearch
MIT License
5 stars 5 forks source link

Elasticsearch - instructions to configure retention #39

Open wioxjk opened 2 years ago

wioxjk commented 2 years ago

Hi!

As someone that is not really an Elasticsearch-guru - setting up retention for the Haraka-logging in elasticsearch is quite a challenge.

I following the very clear instructions here, on about setting up the logging towards the Elasticsearch-server here: https://haraka.github.io/plugins/log.elasticsearch/

But I am really struggling with configuring automatic deletion of logs after X amount of days.

How did you guys solve it? I was made aware of this when elasticsearch was full (999/1000 shards) and therefore was not logging correctly. I "fixed" this by increasing the shards on the installation.

msimerson commented 2 years ago

I solved this for myself by first: rolling up the existing shards into monthly shards instead of daily, and then configuring fewer shards per file.

msimerson commented 2 years ago

I have created a FAQ with some strategies I've used to manage my ES cluster.