search

edenhill / kcat

Generic command line non-JVM Apache Kafka producer and consumer
Other
5.46k stars 484 forks source link

Stream Nginx Logs to Kafka via kcat | messages are not sent as logs are written #445

Open moatazsalem2 opened 8 months ago

moatazsalem2 commented 8 months ago

Hello,

I have started Nginx with access logs being directed to stdio /usr/sbin/nginx -g 'daemon off;'

then updated the command to pipe the access logs to kcat /usr/sbin/nginx -g 'daemon off;' | /usr/bin/kcat -b $KAFKA_HOST -t $KAFKA_TOPIC -P -X queue.buffering.max.ms=1 -l -X batch.num.messages=1 -X batch.size=1 -X queue.buffering.max.messages=1 -X queue.buffering.max.kbytes=1

As you can tell, i have played with many options, but still, out of each 5 page views, i get 4 entries in the topic which indicates that kcat is still buffering some access logs information.

Appreciate your support.

OneCricketeer commented 1 month ago

I'd recommend using something like Fluent-bit for this