search

twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Allow users to use ListShards to check Kinesis stream readiness #43

Closed StevenMcVicker closed 8 months ago

StevenMcVicker commented 5 years ago

This addresses #42. Adds a boolean configuration field for toggling DescribeStream and ListShards in the kenesisStreamReady method that is called in Run

This might not be an ideal solution, but just putting it out there for consideration.

garethlewin commented 5 years ago

Why make this a toggle and not replace DescribeStream directly with ListShards?

garethlewin commented 5 years ago

Hmm, after reading this why do you propose using ListShards to tell if the stream is ready? Is the assumption that ListShards will return ResourceInUseException if kinesis is not ready?

connormckelvey commented 5 years ago

@GarethLewin, I worked with @StevenMcVicker on this. We chose to do a toggle just to try to keep everyone happy. Happy to remove it.

We chose ListShards mainly because it seems to have the highest allowed TPS (100 tps) of any of the APIs so it would hopefully mean running into the LimitExceededException less often.

And yes, the idea is that ResourceInUseException means the stream is not in an active state. This isn't one for one with describe stream, as kinesisStreamReady allows for an UPDATING state as well. From my investigation it looks like getting the StreamStatus is only possible through DescribeStream.

slydon commented 8 months ago

travis is broken, I reviewed and tested this PR in my environment