search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Remove DescribeStream from startup. #67

Closed ThisWillGoWell closed 8 months ago

ThisWillGoWell commented 11 months ago

Remove the call to kinesisStreamReady(). The DescribeStream endpoint has 10 RPS per account, and the alternative DescribeStreamSummary is 20 RPS per account. This can prevent new readers from starting up due to a thundering heard. Since there is no easy way to check if the stream is ready that is not limited (even in #43 there is a upper limit of 100 RPS on ListShards), we should just remove this check.

If a stream is not ready then the reader will just return an error when it goes to read the shards.

slydon commented 8 months ago

Closing as a duplicate of https://github.com/twitchscience/kinsumer/pull/43