Fluentbit stops sending logs to cloudwatch after some time.

Sangeethavisa commented 1 year ago

Describe the question/issue

I'm facing issue with forwarding my application logs to cloudwatch. It does send logs but after some time the connection gets lost between container and cloudwatch. And when I exec the fluentbit.config file again, remaining logs gets flushed to the Cloudwatch. I want all the logs to be read and pushed to the cloudwatch stream without any delay or manual execution.

Configuration

[fluentbit.txt] (https://github.com/aws/aws-for-fluent-bit/files/10015640/fluentbit.txt)

Fluent Bit Log Output

fluentbit_debug_log.txt

Cluster Details

We are having an ECS cluster in which fluentbit container runs as a separate service with a container. This container has been mounted with the EFS ( which has all the applications logs stored in it )

Steps to reproduce issue

Build a custom fluent-bit image and launch a container which has EFS ( contains all app logs ) mounted in a path (say /log ) Dockerfile.txt

Observe in cloudwatch stream for latest logs.

@PettitWesley Please suggest on this, as this is affecting our prod environment.

Thanks!!

zwj102030 commented 1 year ago

Hi could you confirm on which type plugin you are using ? If you are using C plugins we suggest you switch to go-plugin at this moment. Here is detail

PettitWesley commented 1 year ago

Please generally check our debugging guide: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#how-do-i-tell-if-fluent-bit-is-losing-logs

And as already noted, if you are seeing that Fluent Bit just freezes sending to CW without a clear error message/explanation, then try migrating to the Golang cloudwatch plugin. See the comment above.

PettitWesley commented 1 year ago

Please see the note about the cloudwatch hang issue here: https://github.com/aws/aws-for-fluent-bit/issues/542