fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.57k stars 1.53k forks source link

how to minikube's system logs share in opensearch via fluentbit? #9015

Open Ommkwn2001 opened 1 month ago

Ommkwn2001 commented 1 month ago

So i have a three helm charts opensearch, opensearch dashboard and fluentbit.

so when i try to minikube logs pass in the opensearch via fluentbit, there are give me an error in the fluentbit logs and my fluentbit version is 3.0.7 .

The error is :

[2024/06/27 05:32:52] [error] [upstream] connection #68 to tcp://10.244.0.83:9600 timed out after 10 seconds (connection timeout) [2024/06/27 05:32:52] [error] [upstream] connection #72 to tcp://10.244.0.83:9600 timed out after 10 seconds (connection timeout) [2024/06/27 05:32:52] [ warn] [engine] failed to flush chunk '1-1719466229.993033463.flb', retry in 87 seconds: task_id=601, input=cpu.0 > output=opensearch.0 (out_id=0) [2024/06/27 05:32:52] [ warn] [engine] failed to flush chunk '1-1719466254.994056245.flb', retry in 49 seconds: task_id=626, input=cpu.0 > output=opensearch.0 (out_id=0)

when i install fluentbit via helm i pass also this config code The code is: `config: inputs: | [INPUT] Name cpu Tag cpu

outputs: | [OUTPUT] Name opensearch Match * Host opensearch-cluster-master-headless.default.svc.cluster.local Port 9600 HTTP_User admin HTTP_Passwd TadhakDev01 Suppress_Type_Name On Trace_Error On Trace_Output On Logstash_Format On Replace_Dots On Retry_Limit 100 Index minikubelogs `

And I edit this below code in my opensearch's value.yaml file:

ssl: http: false
allow_unsafe_democertificates: true

patrick-stephens commented 1 month ago

All this is saying is that the networking is not right. I think you need to debug that to confirm what is going wrong.

Ommkwn2001 commented 3 weeks ago

when i try to use data prepper, so there are all logs get in opensearch dashboard through flyintbit

patrick-stephens commented 3 weeks ago

Your fluent pid pod cannot talk to that IP you have configured - as to why you will need to investigate.