Reduces the aggregated Kinesis record size from 1 MB to 25 KB
Replaces PutRecord call with PutRecords
Motivation and Context
Kinesis Data Streams charge based on PUT payload units, which have a minimum size of 25 KB. Historically the project aggregated up to the Kinesis limit of 1 MB because there was no support for the PutRecords API, but as of v1.0 there is. There's little (or no) value in sending aggregated records larger than 25 KB and by sending them via PutRecords the throughput performance increases by ~33% to ~50% (see screenshot).
How Has This Been Tested?
This was tested on a high-volume, production data pipeline.
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
PutRecordcall withPutRecordsMotivation and Context
Kinesis Data Streams charge based on PUT payload units, which have a minimum size of 25 KB. Historically the project aggregated up to the Kinesis limit of 1 MB because there was no support for the
PutRecordsAPI, but as of v1.0 there is. There's little (or no) value in sending aggregated records larger than 25 KB and by sending them viaPutRecordsthe throughput performance increases by ~33% to ~50% (see screenshot).How Has This Been Tested?
This was tested on a high-volume, production data pipeline.
Types of changes
Checklist: