I'm using image fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-cloudwatch-1.3 and I also tried with the latest one, but I'm getting this error while trying to log to Cloudwatch.
#<Thread:0x00007f7b44f9b2c8 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:323 run> terminated with exception (report_on_exception is true):
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:382:in `put_events_by_chunk': Log event in <LOG_GROUP_NAME> is discarded because it is too large: 671770 bytes exceeds limit of 262144 (Fluent::Plugin::CloudwatchLogsOutput::TooLargeEventError)
from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:326:in `block (2 levels) in write'
I know this is an issue with the Cloudwatch API that limits the size of the events. Is this fixed for other versions of that image? I've seen that for fluent-bit, the solution was to truncate the log if it's too big.
Any idea on how to fix it for this specific image?
Problem
I'm using image fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-cloudwatch-1.3 and I also tried with the latest one, but I'm getting this error while trying to log to Cloudwatch.
I know this is an issue with the Cloudwatch API that limits the size of the events. Is this fixed for other versions of that image? I've seen that for fluent-bit, the solution was to truncate the log if it's too big.
Any idea on how to fix it for this specific image?
...
Steps to replicate
This is my current config:
Reproducing is kinda difficult, but basically make a container log a very large log entry.
Expected Behavior or What you need to ask
I would expect the plugin to truncate the event or split it in different events.
Using Fluentd and CloudWatchLogs plugin versions
fluentd --version
ortd-agent --version