Retire ES logs & configure cerebro / elasticsearch-kopf

[rasheedamir]

• [ ] Elasticsearch for logs; automatically retire old indices : Curator
• [ ] Configure kops to visualize indices https://github.com/lmenezes/elasticsearch-kopf or https://github.com/lmenezes/cerebro

[hazim1093]

Additional Information: Installing curator via pip:

pip install elasticsearch-curator

Curator requires two files, a configuration file and an action file

References: https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/configfile.html https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/actionfile.html

sample action file to delete indices: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html

Possible solution: Create a cron job or similar which executes curator after a certain period of time

[hazim1093]

One possible sequence of actions would be to delete indices older than e.g. 60 days and close indices older than 30 days

Reference: http://www.ragingcomputer.com/2014/02/removing-old-records-for-logstash-elasticsearch-kibana (uses old version of curator)

[rasheedamir]

Old ES log indices should be purged or moved to S3

Read this for understanding:

• https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html

And then lets choose whichever is easier and better among them:

• https://medium.com/@rcdexta/periodic-snapshotting-of-elasticsearch-indices-f6b6ca221a0c
• http://www.tothenew.com/blog/running-curator-in-docker-container-to-remove-old-elasticsearch-indexes/
• https://alexandreesl.com/2017/01/07/curator-implementing-purge-routines-on-your-elasticsearch-cluster/
• https://github.com/stakater/elasticsearch-logstash-s3-backup
• https://github.com/kayrus/elk-kubernetes#removing-old-indices