search

jasonish / evebox

Web Based Event Viewer (GUI) for Suricata EVE Events in Elastic Search
https://evebox.org/
MIT License
414 stars 67 forks source link

Old events not properly cleared #285

Closed federico-zangani closed 4 months ago

federico-zangani commented 10 months ago

Hi,

I'm using Evebox with the SQLite database, and with no modification I still see events from more than 7 days ago

I tried implementing a size constraint on the database and it seems to be working fine, but the older events still aren't being cleared

Is there a way to fix this?

jasonish commented 10 months ago

What version are you using? If you add -vv you should get some extra debug information and should see something like this once a minute:

2023-09-14 04:23:10 TRACE evebox::sqlite::retention: Deleting events older than 7 days
2023-09-14 04:23:10 DEBUG evebox::sqlite::retention: Deleted 39 events older than 2023-09-07 4:23:10.810783427 +00:00:00 (7 days) in 9 ms
federico-zangani commented 10 months ago

I'm running version 0.17.2 in a docker container, I see

INFO evebox::sqlite::retention: Database retention settings: days=7, size=0 DEBUG evebox::sqlite::retention: Deleted 1000 events older than 2023-09-07 7:40:22.656996778 +00:00:00 in 2661 ms

I assume my config file is not being read because the size constraint that I put is not there and was probably just matched by chance when I checked, but in the web page I still see older events, aswell as some alerts dating more than 7 days

immagine

immagine

Thanks for the help

jasonish commented 9 months ago

For the config file, make sure your Docker volumes are correct and you are specifying the path to the configuration file. if you still run into issues there, let me know. But this is how I run EveBox as well.

It may take a while to delete all the old events. Write activity in SQLite locks the database, so we only delete 1000 events at a time. If 1000 events were actually deleted, we reschedule the next purge in 1 second rather than 1 minute. So if you are dropping from a retention of 30 days to 7, in can take a while. There is also an issue if you happen to be ingesting more than 1000 events per second.

federico-zangani commented 9 months ago

I specified the config file in a better way and it's now working properly

I went into the debug logs to see and it seems like it is now slowly deleting old events, but since I get more than 1000 per second at some times they still showed up

I'm also experiencing some lag (evebox loads events very slowly and falls behind compared to the suricata logs), could those be related?

Thanks :)

jasonish commented 4 months ago

Looks like this issue is resolvd. Detecting lag would be a good to add tho.