search

fluent / fluent-bit-kubernetes-logging

Fluent Bit Kubernetes Daemonset
Apache License 2.0
468 stars 248 forks source link

fluent-bit kafka data pipeline is not working. #100

Open bharathreddy901 opened 1 year ago

bharathreddy901 commented 1 year ago

Hi Team,

We have configured the fluent-bit->Kafka data pipeline with the below mentioned link.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

When we have deployed this pipeline, pods are coming up, Kafka topic was created, but logs are not flowing to Kafka cluster. We could not see any error in the pod logs, we are seeing below info. #########################################################

[root@centos]# kubectl logs fluent-bit-nb886 -n ec2-kafka Fluent Bit v1.5.7

[2022/09/12 07:11:40] [ info] [engine] started (pid=1) [2022/09/12 07:11:40] [ info] [storage] version=1.0.5, initializing... [2022/09/12 07:11:40] [ info] [storage] in-memory [2022/09/12 07:11:40] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443 [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] local POD info OK [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server... [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK [2022/09/12 07:11:40] [ info] [output:kafka:kafka.0] brokers='x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092' topics='kafka-bit' [2022/09/12 07:11:40] [ info] [sp] stream processor started ##########################################################################

Please let us know how to solve this issue.

Thanks in advance..!!

bharathreddy901 commented 1 year ago

Hi Team - Please let me know if anyone resolved above problem.

bharathreddy901 commented 1 year ago

Hi @edsiper and @hobti01,

Logs are not flowing from Fluent-bit to Kafka, But Kafka topic is getting created. We are using below link to configure the pipeline. We have not changed anything, only thig we have updated Kafka broker name & topic name.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

Thanks in advance.