It seems that all of our customers probably are using log_key very simply to send string log lines that were wrapped in a simple json object when Fluent Bit ingested them, and so they set log_key log. And that works. But besides that case, log_key appears to be mostly broken.
If the value of the log_key is a numeric value, Fluent Bit only sends one digit to CW for some reason. For example, I sent 666 and in CW I only see 6 Once. If the value of log_key is a map object, then the upload fails since this makes the serialized request payload invalid somehow:
[2022/06/14 18:10:07] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2022/06/14 18:10:07] [ warn] [engine] failed to flush chunk '950-1655230206.669932363.flb', retry in 8 seconds: task_id=0, input=forward.0 > output=cloudwatch_logs.0 (out_id=0)
[2022/06/14 18:10:15] [error] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents API responded with error='SerializationException'
The
cloudwatch_logs
plugin has alog_key
option which sends the value of the given key to cloudwatch instead of the whole record. There is some discussion of log_key here: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/cloudwatchlogsIt seems that all of our customers probably are using log_key very simply to send string log lines that were wrapped in a simple json object when Fluent Bit ingested them, and so they set
log_key log
. And that works. But besides that case,log_key
appears to be mostly broken.I used the fluent python logger to test sending different logs: https://github.com/fluent/fluent-logger-python
If the value of the log_key is a numeric value, Fluent Bit only sends one digit to CW for some reason. For example, I sent
666
and in CW I only see 6 Once. If the value of log_key is a map object, then the upload fails since this makes the serialized request payload invalid somehow: