add kafka as a new logging backend

jmpsec / osctrl

Fast and efficient osquery management

https://osctrl.net

MIT License

391 stars 51 forks source link

add kafka as a new logging backend #524

Closed saryani closed 1 month ago

saryani commented 1 month ago

This PR will make it possible for osctrl to send the osquery-agent logs to a kafka topic. this means that these logs will no longer be pushed to the database, you need to set kafka as the log backend at init time otherwise it will log into database.

added a logger backend to send the agents logs to kafka
added required configuration definition to allow setting up a kafka logger
updated the docker-compose to allow to run different backends

saryani commented 1 month ago

sorry @javuto I meant to make this PR agains my fork of osctrl before making it public, please ignore this PR, I will push the the final PR once we did some more tests, cheers

javuto commented 1 month ago

No problem, the only feedback I had after a quick look, is probably to avoid adding any Kafka related elements to the docker development files, since is not really part of the stack to do local development. Thank you!