search

awslabs / amazon-kinesis-producer

Amazon Kinesis Producer Library
Apache License 2.0
400 stars 331 forks source link

Throughput slows down and halts completely when using more than 8 shards #343

Open albehrens opened 3 years ago

albehrens commented 3 years ago

I achieve most performant throughput when my Kinesis stream has 8 shards. Throughput slows down when using 10 shards and does not deliver any records anymore when using 16+ shards. How can this be?

I set following properties because I do not want to lose any records:

RecordMaxBufferedTime = 60000
RecordTtl = 300000000000

Can this have anything to do with it?

VladimirPchelko commented 2 years ago

The value of RecordMaxBufferedTime is too high.

When you increase the number of shards - you proportionally increase the number of records - that KPL waits for the buffer to fill before publishing data to Kinesis - most likely the buffer stops filling completely, and KPL always waits for RecordMaxBufferedTime

In most cases, RecordMaxBufferedTime value greater than a second severely reduces throughput.