search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

kenesisStreamReady produces LimitExceededException when calling DescribeStream #42

Open StevenMcVicker opened 5 years ago

StevenMcVicker commented 5 years ago

We are experiencing very frequent LimitExceededExceptions when starting our kinsumer applications. This is a result of the hard 10 TPS rate limit on DescribeStream calls in AWS.

Right now we are avoiding this by creating a kinsumer with the NewWithInterfaces constructor and passing in a custom kinesisiface.KinesisAPI with a workaround implementation of DescribeStream that calls ListShards instead.

It would be great if there was a config toggle to use an alternate method to check kenesis stream readiness so that we can avoid this issue.

garethlewin commented 5 years ago

Hi, @StevenMcVicker I'll take a look soon, but are you using the latest? I thought that https://github.com/twitchscience/kinsumer/pull/21 addressed this (if it doesn't I'm just going by the description of the issue and it might not be related)

garethlewin commented 5 years ago

Oh, I see #21 didn't replace all uses of DescribeStream.