democratech / 2022

4 stars 2 forks source link

Forward the Kibana and ElasticSearch logs to Fluent-bit #51

Open JMLX42 opened 3 years ago

ThomasVille commented 3 years ago

@JMLX42 I don't quite understand the title of this issue.

Usually, FluentBit collects the logs from other services, stores them in Elasticsearch and Kibana is used to visualize these logs and create dashboards.

Is the goal of this issue to actually forward the logs of Kibana and ElasticSearch to FluentBit, which will in turn store them in ElasticSearch and allow us to visualize them using Kibana?

Resources:

ThomasVille commented 3 years ago

Actually, I can see that the other services are already configured to send their logs to FluentBit, so I guess the goal of this issue is really to forward the logs of Kibana and ElasticSearch to FluentBit. But wouldn't this create an infinite loop where:

  1. Ghost sends its logs to FluentBit.
  2. FluentBit sends them to ElasticSearch.
  3. ElasticSearch sends logs to FluentBit.
  4. FluentBit sends them to ElasticSearch.
  5. ElasticSearch sends logs to FluentBit.
  6. ...

Maybe there's a way to avoid this though.

JMLX42 commented 3 years ago

But wouldn't this create an infinite loop

Good question. I don't know if ElasticSearch and Kibana show logs for every query...

JMLX42 commented 3 years ago

@ThomasVille TBH I don't even know if it's such a good idea. The other EFK stacks I saw did not forward the ElasticSearch and Kibana logs to Fluent-bit IIRC.