cloudwatch_logs log_key only works with string values

The cloudwatch_logs plugin has a log_key option which sends the value of the given key to cloudwatch instead of the whole record. There is some discussion of log_key here: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/cloudwatchlogs

It seems that all of our customers probably are using log_key very simply to send string log lines that were wrapped in a simple json object when Fluent Bit ingested them, and so they set log_key log. And that works. But besides that case, log_key appears to be mostly broken.

I used the fluent python logger to test sending different logs: https://github.com/fluent/fluent-logger-python

If the value of the log_key is a numeric value, Fluent Bit only sends one digit to CW for some reason. For example, I sent 666 and in CW I only see 6 Once. If the value of log_key is a map object, then the upload fails since this makes the serialized request payload invalid somehow:

[2022/06/14 18:10:07] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2022/06/14 18:10:07] [ warn] [engine] failed to flush chunk '950-1655230206.669932363.flb', retry in 8 seconds: task_id=0, input=forward.0 > output=cloudwatch_logs.0 (out_id=0)
[2022/06/14 18:10:15] [error] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents API responded with error='SerializationException'

aws / aws-for-fluent-bit

cloudwatch_logs log_key only works with string values #367