search

SUSE / linux-security-sensor

Linux security sensor
Other
18 stars 9 forks source link

Add Kafka-Humio Gateway [Depends on PR#10] #8

Closed jeffmahoney closed 2 years ago

jeffmahoney commented 2 years ago

This PR adds a Kafka plugin that can be used to export events to a Kafka instance. The instance is expected to be running in the same container ecosystem as the velociraptor server. We do this so that we can ensure that events are queued properly if the Humio server can't be reached. Once the events are queued, a consumer formats and forwards them to Humio.

jeffmahoney commented 2 years ago

I think the windows failure is a testcase issue. There's a timer that trips after 2 minutes and dumps state. It's reset for each test. These changes don't touch anything that would be involved there.

lorddoskias commented 2 years ago

Overall it looks good I only have a couple of minor nits which I've put inline. For an initial version I think it's good to go.

jeffmahoney commented 2 years ago

On average how many messages would be in a single claim? I was wondering if we could do the batching on claim granularity i.e not have sendEvent and the eventChannel but simply buffer all mesages from a single claim and make that be the batch. I guess we could end up in situation where we have multiple claims consisting of a single message? Anyways this is just a remark and the changes LGTM

The Claim isn't a one-shot. It starts up a goroutine that feeds messages from Kafka from each claim until the context is canceled or the server rebalances (and then it'll just reconnect). That's also why sendEvents is a separate goroutine -- we want to push events on both a timeout and a batch limit, and we can't do that if we're waiting on the for loop in ConsumeClaim.