Changes the payload size limit for Kinesis Data Streams from MiB to MB
Motivation and Context
Discovered an edge case where the system generated a Kinesis Data Stream payload of more than 5 MB when using the PutRecords API. Changing the limit of each record from 1 MiB (1048576 bytes) to 1 MB (1000000 bytes) fixed the problem. According to the AWS docs, Firehose uses 1 MiB as its message size limit.
How Has This Been Tested?
Tested E2E in the production data pipeline the bug was discovered in.
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Motivation and Context
Discovered an edge case where the system generated a Kinesis Data Stream payload of more than 5 MB when using the PutRecords API. Changing the limit of each record from 1 MiB (1048576 bytes) to 1 MB (1000000 bytes) fixed the problem. According to the AWS docs, Firehose uses 1 MiB as its message size limit.
How Has This Been Tested?
Tested E2E in the production data pipeline the bug was discovered in.
Types of changes
Checklist: