search

rocknsm / rock

Automated deployment scripts for the RockNSM network hunting distribution.
http://rocknsm.io
Apache License 2.0
446 stars 96 forks source link

Centralized dataflow support #71

Open dcode opened 7 years ago

dcode commented 7 years ago

This should be relatively easy to implement with Logstash aggregating, but I've had a couple people ask lately for an ability to setup a clustered mode similar to that of Security Onion. Just dropping here for idea recording.

charlesfair commented 7 years ago

So, clustering via Bro workers sending to a centralized manager running Bro manager. That would be quite straightforward, though "different" from Securityonion's model. If done that way, sensor setup could be straightforward done through the yaml file prior to starting installation. Also, could setup a Bro logger and proxy if needed because of the architecture. This would be better than Security Onion architecture because each sensor could be clustered for redundancy if desired, not possible with SO.

charlesfair commented 7 years ago

Untested, but could be valuable if it would work, Kafka could be used between the Bro workers and the manager.

charlesfair commented 7 years ago

From Doug Burks on Securityonion "built-in" clustering capability:

`Start with one Security Onion box, run Setup, choose Advanced Setup, and choose Master. This box will be a Master only, no sniffing. Then build your separate sensor boxes, choosing Advanced Setup, and then choosing Sensor. Sensors will check into the Master server. Then you use flow based load balancing to split your large traffic volume up amongst the sensors. When you log into Snorby/Squert/Sguil/ELSA, you will see alerts/logs from all sensors at once.

As far as Bro clustering, Advanced Setup puts Bro into cluster mode. Each sensor is its own Bro cluster by default (manager, proxy, and one or more Bro workers). All Bro logs go into the local ELSA database on the sensor itself, but when you log into the central ELSA web interface it queries all ELSA sensor databases in parallel, showing you all Bro logs across your entire deployment.

Doug Burks

dcode commented 7 years ago

Yeah, we're not going to support clustering of multiple bro workers across nodes. That capability is all-too-often abused. We'll catch the data upstream and merge it into the same topic. Also, Kafka doesn't meet the needs of inter-bro communication. Broker was created for this purpose, but it is not yet ready for prime-time.