brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
330 stars 21 forks source link

fix(kinesis): remove Kinesis aggregation size and count limits #168

Closed shellcromancer closed 6 months ago

shellcromancer commented 6 months ago

Description

Substation uses the internal/aggregate package to buffer events with consideration to buffer size, record counts, and durations which shadow the checks here with adherence to Kinesis limits in a more user friendly way.

Motivation and Context

Inclusion of the size checks here allowed events between 25KiB and 1MiB to be sent as empty events to destinations.

How Has This Been Tested?

The fix has been tested on the internal pipeline containing events ranging from 10KiB to 1MiB with no issues.

Types of changes

Checklist: