Detection of no-report from Fluentbit

[atxryan]

Per @kforeverisback: "We followed these two documentation which directly addresses the buffering problem

1. Buffering & Storage - Fluent Bit: Official Manual - https://docs.fluentbit.io/manual/administration/buffering-and-storage
2. Backpressure - Fluent Bit: Official Manual - https://docs.fluentbit.io/manual/administration/backpressure

I concur that it’s not the real problems we were facing, but in case on high loads and apparent log-analytics service availability this buffering can come in handy."

[jomalsan]

@atxryan,

@chkinney and I are interested in working on this issue as we think it will be good for upskilling on the project. Our interpretation is that we want to have a doc outlining good habits when working with Fluent Bit and going over a few decision points that users may run into.

Looks like the two topics above are good candidates, but we are wondering if there are other areas people have run into questions

[jomalsan]

Spoke with @kforeverisback:

Real issue is that FluentBit failed silently while Kushal was Point Dev and the changes above (Buffering, Back Pressure) seemed to fix them. He is unsure if there are other good habits to use when working with FluentBit

Other context:

• FluentBit is the only way that we are getting logs to Log Analytics, so it is hard to detect when FluentBit is not reporting logs
• Prometheus and Grafana are being added to the cluster now

Proposed Actions:

• Test the built-in FluentBit prometheus metrics and grafana dashboard for monitoring the FluentBit pod
• Alert in Azure due to a lack of logs from a cluster?
• Troubleshooting doc that includes:
  • Example query for an alert
  • Steps for setting up the FluentBit dashboard in grafana
  • Troubleshooting section on problems we have seen and how to address them (If FluentBit docs have a trouble shooting section then just link to that)