search

orchestructure / presentations

Holds and organizes all past, present, and future presentations at the meetup
http://orchestructure.io/
41 stars 11 forks source link

Lightning Talk Idea: Log Collection - Fluent-bit vs Fluentd vs Filebeat : FIGHT! #26

Closed StevenACoffman closed 6 years ago

StevenACoffman commented 6 years ago

Fluent-bit rocks

A short survey of log collection options and why you picked the wrong one. 😜

Log Collection

Principle 11 of the 12 Factor App is to "Treat logs as event streams".

While most traditional applications store log information in a file, the Twelve-Factor app directs it, instead, to stdout as a stream of events; it’s the execution environment that’s responsible for collecting those events. That might be as simple as redirecting stdout to a file, but in most cases it involves using a log router such as Fluentd, Filebeat, or Fluent-bit and saving the logs to Hadoop or a service such as Splunk.

In docker, the default log driver is json-file, but it also supports others, such as fluentd. Collection and shipping is otherwise bring your own.

In Kubernetes, you have at least two battle tested choices for automatic logging capture: Stackdriver Logging if you’re using Google Cloud, and Fluentd to Elasticsearch if you’re not. Both of those are actually Fluentd, since Stackdriver Logging uses a Google-customized and packaged Fluentd agent. You can find more information on setting Fluentd Kubernetes logging destinations here.

Filebeat is more common outside Kubernetes, but can be used inside Kubernetes to produce to ElasticSearch.

Fluent-bit is a newer contender, and uses less resources than the other contenders.

Why Fluent-bit rocks:

Resource Comparison

Without monitoring to tailor to our workloads, just going from the recommended resource requests and limits, we have a stark contrast between the different logging collection.

Beats vs Logstash:

Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Beats have a small footprint and use fewer system resources than Logstash.

Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.

Fluent-bit vs Fluentd:

Fluentd and Fluent Bit projects are both created and sponsored by Treasure Data and they aim to solves the collection, processing and delivery of Logs.

Both projects share a lot of similarities, Fluent Bit is fully based in the design and experience of Fluentd architecture and general design. Choosing which one to use depends of the final needs, from an architecture perspective we can consider:

Fluentd is a log collector, processor, and aggregator. Fluent Bit is a log collector and processor (it doesn't have strong aggregation features such as Fluentd).

Combinations

Fluent-bit or Beats can be a complete, although bare bones logging solution, depending on use cases. Fluentd or Logstash are heavier weight but more full featured.

You can combine Fluent-bit (one per node) and Fluentd (one per cluster) just as you can combine Filebeat (one per node) and Logstash (one per cluster).

Comparisons

Fluent-bit from this file

        resources:
          requests:
            cpu: 5m
            memory: 10Mi
          limits:
            cpu: 50m
            memory: 60Mi

Fluentd from this file:

        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi

FileBeat from this file:

        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi

Keeping Stacktraces together

Most programs contain bugs, and those lead to valuable multi-line stacktraces which are unpleasant to reassemble after being shipped to an eventually consistent distributed data sink (ElasticSearch, Kafka, AWS S3, DynamoDB, what-have-you). It is more convenient if the collector could understand and keep those as single messages.

In fluentd, this is accomplished through fluent-plugin-detect-exceptions which has artisanally hand-crafted regexes for most languages.

In fluent-bit, you configure a multi-line parser for each language you wish to support, and have your application add an annotation that hints what parser to use. Feel free to steal regexes from the fluentd plugin above.

Resilience and Reliability

In kubernetes, using the default docker json-file log driver already provides a measure of on disk buffering for ephemeral containers. When Fluent-bit is tailing those files, it the recommended option is to use a sqlite database file can be used so the plugin can have a history of tracked files and a state of offsets. This is very useful to resume the state if the service is restarted. You may specify a retry limit for shipping logs to different outputs (including False which will retry forever).

In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restrict the amount of data than an input plugin can ingest, this is done through the configuration parameter Mem_Buf_Limit.

Monitoring

Prometheus Metrics out of the box in the 0.13.x series! Woohoo!

Log Pipeline

Ok great, we're collecting and shipping... and then what? If you want to do more than just searching ElasticSearch, you might consider a solution like minipipe to enable sophisticated analytics.

At Ithaka, here's a presentation about what our Log Pipeline and Analytics stack look(ed) like

InAnimaTe commented 6 years ago

Wow, awesome breakdown. As long as your lightning talk is no longer than 5 minutes (the time from when you start talking to the time you stop talking, not including taking questions or later side conversations with other members), then I approve your doing this at the upcoming meeting on the 28th!

StevenACoffman commented 6 years ago

Thanks, I plan on only listing the "why fluent-bit bullet points" with maybe an introduction sentence or two on the 12-factor app collection principle. I can refer people to this issue if they want to learn more.

InAnimaTe commented 6 years ago

Perfect! I'm going to post it on twitter and edit the Meetup description with a link to it as well.

InAnimaTe commented 6 years ago

Ok posted links so people can easily find details. Going to close this.

NeckBeardPrince commented 2 years ago

@StevenACoffman I know this is coming from the grave, but I wondered how accurate is this in 2021.

StevenACoffman commented 2 years ago

Perfectly accurate. It is exactly what was true in 2018. 🤣

Honestly, it is still pretty accurate, but you want to use the latest versions of everything, and so the config is slightly different. fluent-bit has continued to improve while both fluentd and filebeat have not changed much.