search

StamusNetworks / scirius

Scirius is a web application for Suricata ruleset management and threat hunting.
GNU General Public License v3.0
614 stars 150 forks source link

Ability to change the Logstash indice names #221

Closed ipworkx closed 3 years ago

ipworkx commented 3 years ago

Hi,

I have come to the issue that the amount of indices became too large because Logstash created an indice per event, per day. I know curator deletes indices when they become too old. But what if we want to decide to save the indices for at least half a year? Further on the indices become very small. Official an indice should become around 30-40 GB per indice to make a right balance between memory, speed and performance when searching. So it's better to configure logstash to save the indices with just a name without rolling per day. This could be done by changing the indice roll over per month instead per day. Or just by using just one indice for events and just one indice for stats.

If I do this and also change the "local_settings.py" file according the altered logstash index name, scirius does not shows alerts anymore. It seems that Scirius wants to have a separate "Logstash-alert" indice name with a date embedded within the name.

I would like to see configuration parameters to be able to change or configure an "alert" indice, an "events" indice and a "stats" indice. It would be nice to do the roll-over within elastic itself. In this way the end user is in control to save the amount of event data without breaking the boundary of 1000 shards (1 shard per indice) when reaching to the point of an half year later.

Regards, Thierry

ipworkx commented 3 years ago

I found out this is practically impossible without modifying the scirius application. Within the code there are somewhere calculations regarding the use of year/month/day. This is needed to look at the indicename. I'll close this request.

pevma commented 3 years ago

Scirius uses the logstash-alert-* index to display alerts as it queries based on that ES (and time/date as it is less performance intensive on the queries). You can of course change that behavior but i wonder if you do not roll over every day - in most deployments if the indexes are going to grow simply huge.

If you decide to save the indices for over half a year - you could adjust the curator job to clean up only older than 6mo - would that work ?

ipworkx commented 3 years ago

The issue is if the index as named like logstash-alert-2021.05.12 it works. But for instance if I name the index logstash-2021.05 it failed to show up the alerts in scirius. I did see something with the last part of the index name. It is sort of build in unless I change the code itself.

For rollover I prefer ILM.

pevma commented 3 years ago

Yes, that Scirius setting - the alert index - is set in form /etc/scirius/local_settings.py