There's an edge case where an empty aggregated Kinesis record can be sent if a single record is between ~0.9999 MB and 1 MB. Aggregated Kinesis records have a format and contain more data than a naked record, so there is a small chance that the naked record cannot actually fit inside an aggregated record and be successfully sent to the Kinesis service (due to the 1 MB record service limit).
This record is silently dropped, but since this causes an issue identical to this, it may be worth raising the same error for this edge case.
How Has This Been Tested?
This is running on some high-volume production data pipelines that are known to have this edge case.
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Motivation and Context
This is a fix that is in addition to https://github.com/brexhq/substation/pull/168.
There's an edge case where an empty aggregated Kinesis record can be sent if a single record is between ~0.9999 MB and 1 MB. Aggregated Kinesis records have a format and contain more data than a naked record, so there is a small chance that the naked record cannot actually fit inside an aggregated record and be successfully sent to the Kinesis service (due to the 1 MB record service limit).
This record is silently dropped, but since this causes an issue identical to this, it may be worth raising the same error for this edge case.
How Has This Been Tested?
This is running on some high-volume production data pipelines that are known to have this edge case.
Types of changes
Checklist: