search

cisagov / LME

Logging Made Easy (LME) is a no cost, open source platform that centralizes log collection, enhances threat detection, and enables real-time alerting, helping small to medium-sized organizations secure their infrastructure.
https://www.cisa.gov/resources-tools/services/logging-made-easy
Other
842 stars 66 forks source link

Look into Curator #125

Open aarz-snl opened 10 months ago

aarz-snl commented 10 months ago

Because of the difficulty of supporting each and every case when it comes to hard drive size we may want to look into curator:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

It can be installed as a docker container

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/docker.html

You can configure it to detect space usage by indices:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html

Then you create an 'action' which is what curator will perform when the filter requirements are met:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/delete_indices.html

Could be potential here to write "actions" and "filters" that gives us more control over space management than the default elastic lifecycle policy.

aarz-snl commented 10 months ago

Because our 'shard' size is set to 50GB for indexes this at least gives us a good starting point for MUST HAVES. If your hard drive is less than 50GB this type of logic would end up deleting the only index you have. You have to have enough storage to 'rollover' from winlogbeat-00001 to 00002 which happens after 30 days OR 00001 reaches 50GB in size.