search

fluent / fluent-bit-kubernetes-logging

Fluent Bit Kubernetes Daemonset
Apache License 2.0
468 stars 248 forks source link

Best Practice Question: Multiple fluent-bit or single with multiple outputs? #12

Closed StevenACoffman closed 6 years ago

StevenACoffman commented 6 years ago

We need to be able to output (with Retry_Limit false) to:

If one output fails (say ElasticSearch is down), but the others succeed, what happens?

What I am really asking is what is your recommended best practice. I can either have multiple fluent-bits with a single output or a single fluent-bit with multiple outputs. Which would you recommend?

edsiper commented 6 years ago

Retries are applied per output.

If all Retry_Limit = False and only Elasticsearch fails, Elasticsearch is retried by the scheduler, since the other plugins did OK, no retry is required.

About having one Fluent Bit or multiple ones will only depends of the number of data ingested, filters applied and so on. If you have a slow file system and need to use File output, then I would suggest to move that to a different process context.

StevenACoffman commented 6 years ago

Awesome! Extremely helpful. Thanks for saving me loads of time experimenting.