KPL code is not able to PUT more than average of 5 records per seconds to Kinesis Stream

[SoujanyaRama]

We are trying to post every Log message generated from every application to Cloud Elasticsearch service using Kinesis Stream and FireHose Delivery Stream.

In the attempt, we are using KPL code using producer.addUserRecord to PutRecords to Kinesis stream. The issue we are facing is that the KPL code is only able to post average of 5 records per seconds to Kinesis Stream.

Here is the config used. Irrespective of any combination of values tried, we noticed the producer is only able to send average of 5 records per second to Kinesis stream. config.setMaxConnections(24); config.setCollectionMaxCount(100); config.setCredentialsRefreshDelay(100); config.setRequestTimeout(60000); config.setThreadingModel("PER_REQUEST"); config.setThreadPoolSize(100); config.setAggregationEnabled(true); config.setRecordMaxBufferedTime(100);

Could you suggest if there could be any configuration that is causing the bottleneck?

Thanks & Regards, Soujanya

[pgieser]

KInesis has a write limit of 1MB/sec/shard. Perhaps your stream does not have enough shards?

https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html

[cheefoo]

What is your average record size? and are you getting any write provision throughput exceptions.