twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Paginate when calling ListShards #53

Closed teamdoug closed 4 years ago

teamdoug commented 4 years ago

ListShards has a limit of 1000 shards returned. If your stream has more than that, kinsumer will stop processing shards past the 1000th.

Add manual pagination to the call since ListShardsPages does not exist in aws-sdk-go. Updated some comments that were out of date.

I've tested this by running a slightly modified version against a very large stream.