search

idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
353 stars 58 forks source link

allow total index size-based pruning for opensearch-remote and elasticsearch-remote database modes #446

Closed mmguero closed 2 weeks ago

mmguero commented 6 months ago

We do some very basic index management when using the database mode opensearch-local which is basically "how much storage do the indexes take up" and if it exceeds that threshold it calls DELETE on the oldest ones until it drops below the threshold, controlled by these environment variables.

I'm thinking it might be useful for those who want to keep their indexes from exceeding a certain size but don't want to go full-blown down the ILM/ISM route to allow this for elasticsearch-remote and opensearch-remote as well. Need to look into the ramifications of this.

Also, if/when we do this, I'd like to relocate these variables from dashboards-helper to opensearch.env, probably, as I think they make more sense there.

mmguero commented 2 weeks ago

Finished, and tested using opensearch-local and elasticsearch-remote