search

aws / amazon-cloudwatch-logs-for-fluent-bit

A Fluent Bit output plugin for CloudWatch Logs
Apache License 2.0
175 stars 49 forks source link

Is this now deprecated in favour of the vanilla CloudWatch plugin? #116

Closed pranavmarla closed 3 years ago

pranavmarla commented 4 years ago

I see that v1.6 of upstream/vanilla Fluent Bit now includes its own output plugin for writing to CloudWatch, and says this:

This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the aws/amazon-cloudwatch-logs-for-fluent-bit Golang Fluent Bit plugin released last year. ... Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

However, I don't see any corresponding deprecation notice here. So, to clarify:

  1. Is this Amazon-flavour CloudWatch plugin now deprecated?
  2. If not, for new users who want to use Fluent Bit to send logs from EKS to CloudWatch Logs, should we use: a) The Amazon-flavour Fluent Bit Docker image (containing this Amazon-flavour CloudWatch plugin) OR b) The vanilla Fluent Bit Docker image (containing the new vanilla CloudWatch plugin)
PettitWesley commented 4 years ago

@pranavmarla Sorry... I keep forgetting to put a note in this repo about the new core plugins.

First all, this plugin is not being deprecated- I need to update the statement in the core documentation. Basically, the core cloudwatch_logs plugin contains a subset of the features of this plugin (cloudwatch), and you should use cloudwatch_logs if that subset of functionality is sufficient for you. Because its more efficient, higher performance, and can achieve a higher max throughput of logs.

This plugin, cloudwatch, will continue to exist though since it is easier for people to contribute in Go. For example, the stream and group templating feature that David Newhall wrote is awsome, and he could write it in this plugin since he's a Go programmer and since Go had a library which could be used. He couldn't have built that feature in the C plugin since he's not a C programmer.

So both will continue to be supported. Also, both are contained in AWS for Fluent Bit. And we plan to continue to do that. We of cource recommend you use the AWS release of Fluent BIt.

Finally, both plugins were written by me/our team and have support from Amazon.

PettitWesley commented 4 years ago

Action Item: Clarify this in a note in this repo's readme and in the core Fluent Bit documentation.

I think we should also print a statement when this plugin's code starts up reminding people to check out the new one and migrate if they can.

We will keep this one around, but we do want people to migrate to the new/better plugin.

pranavmarla commented 4 years ago

Great, thanks for clarifying @PettitWesley !

pranavmarla commented 4 years ago

@PettitWesley Actually, one more question: As far as I understand, the only difference between the 'AWS for Fluent Bit' Docker image and the regular core Fluent Bit Docker image is that the AWS image is the only one that contains the Go plugins -- is that correct?

i.e. AWS Fluent Bit Image: Contains core cloudwatch_logs plugin and Go cloudwatch plugin

Core Fluent Bit Image: Contains core cloudwatch_logs plugin only

Is that correct?

PettitWesley commented 4 years ago

Is that correct?

Mostly yes. The AWS for Fluent Bit image is also distributed in regional ECR repos. And if you have an AWS Support agreement, they'd require that you use AWS for Fluent Bit if you want help from support.

pranavmarla commented 4 years ago

Great, thanks for your help!

nathanielks commented 4 years ago

Came here to ask all the same. Thanks for all the clarifications!

zhonghui12 commented 3 years ago

A notice has been added to README. This issue will be closed: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/pull/127.