[kinesis-firehose] Support simple_aggregation option to send multiple log events as a record

[kangaechu]

I changed the Firehose plugin from firehose (aws/amazon-kinesis-firehose-for-fluent-bit) to kinesis_firehose (core plugin) in my project, which caused a sharp increase in billing.

Upon investigation, we found that the firehose plugin supports simple_aggregation, so it was sending multiple records as a single event.

"{\"DeliveryStreamName\":\"fluent-bit--test\",\"Records\":[{\"Data\":\"(records encoded by BASE64)\"}]}"

However, kinesis_firehose does not support simple_aggregation, so events are created for each record.

"{\"DeliveryStreamName\":\"fluent-bit--test\",\"Records\":[
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"}]}"

There were plans to implement simple_aggregation when the kinesis_firehose plugin was created, but as of August 2024, it has not yet been implemented.

https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit/issues/12#issuecomment-791648528

Additional context

• https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit/pull/105
• https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit/issues/12

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.