fix(kinesis): Limit Aggregated Records to 1MB - Githubissues

brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

https://substation.readme.io

MIT License

322 stars 16 forks source link

fix(kinesis): Limit Aggregated Records to 1MB #169

Closed jshlbrd closed 5 months ago

jshlbrd commented 5 months ago

Description

Adds a 1MB limit to the Kinesis KPL aggregator

Motivation and Context

KPL records still need a limit, otherwise the Kinesis transform may generate a record bigger than the 1 MB service limit.

How Has This Been Tested?

Integration tested on data that is known to cause this issue.

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.