search

idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
353 stars 58 forks source link

kubernetes (next steps) - workers with multiple replicas/scale out #182

Closed mmguero closed 4 days ago

mmguero commented 1 year ago

Sub issue of #149

We need to look at the processes where a worker takes a piece of work (for example: file scanners take a file from the zeromq topic to scan, filebeat looks at logs to ingest, etc.)... what happens if we have multiple replicas of the containers in which these workers live? if they're all sharing the same volumes (which for things like artifacts and files, they are, right?), then how do two workers not end up duplicating the work?

I feel like this is a little bit later down the line in priority for getting this all to work (everything can work fine now with just one replica, and the biggest piece that needs to scale out is opensearch, which can be handled already with remote instances), but it's something we need to be aware of.

mmguero commented 1 year ago

marking as "next steps" in title as this will be in the next Malcolm Kubernetes feature update, not the initial one. For the initial release the replicas should probably be set to 1.

mmguero commented 8 months ago

A lot of the groundwork for this is being done in idaholab/Malcolm-Helm

mmguero commented 4 days ago

Going to close as a duplicate of #187