search

StamusNetworks / scirius

Scirius is a web application for Suricata ruleset management and threat hunting.
GNU General Public License v3.0
626 stars 150 forks source link

Scirius strange behaviour: No continuous data collection #135

Open larsbru opened 6 years ago

larsbru commented 6 years ago

I have installed a fresh SELKS System an did the Update by script as described in SELKS Handbook.

My system collects only some hours per day... thats strange. Suricata, Elastic, Disk and Memory icons are green. Does someone know how to fix this?

selks

pevma commented 6 years ago

Are the graphs in Kibana continuous or they show similar behavior?

larsbru commented 6 years ago

Yes, exactly the same. As if no traffic comes in in that specific period of time.

pevma commented 6 years ago

Can you check the log files themselves then and see if in there is a gap int the timestamps as well ? If not then most likely it is something related to shipping I guess.

larsbru commented 6 years ago

What logfile do you mean? I found some errors in e.g. "logstash-plain-2018...log" what other files shall i concentrate on?`

These errors are dropped very often (almost every 15minutes....[2018-05-07T00:15:29,123][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"$ [2.....

Maybe my config file is wrong?

I think shipping of the Data should be fine as i also used a virtual machine of Alienvault before to analyse the same datastream... My network traffic is simply mirrored to hat NIC "ens33" by the switch itself.

pevma commented 6 years ago

Can you describe a bit more about your set up - is it ELK5/6 ? The data i meant the data in the logfile that is getting shipped (eve.json) for example - is it continuous there ?

These errors are dropped very often (almost every 15minutes....[2018-05-07T00:15:29,123][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"$

Judging by the above I am guessing there is a problem with the logstash shipping - it has the pipeline stopped processing new events - seems logstash is not shipping. I would start troubleshooting form there.