search

harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
264 stars 90 forks source link

getShardIDs returns 100 shards at max #80

Closed lordfarhan40 closed 5 years ago

lordfarhan40 commented 5 years ago

getShardIDs use "DescribeStream" just once which in turn only return 100 shards at max. We need to call DescribeStream a few times to get all the shards, something like this Retrieving Shards Reference: https://github.com/aws/aws-sdk-go/blob/master/service/kinesis/api.go#L3891

harlow commented 5 years ago

nice catch, so it looks like ListShards supports pagination

{
   "ExclusiveStartShardId": "string",
   "MaxResults": number,
   "NextToken": "string",
   "StreamCreationTimestamp": number,
   "StreamName": "string"
}

Which means that we could loop over the pages of shards

lordfarhan40 commented 5 years ago

Yes, we can iterate over the response and keep adding the shards. I will start working on the fix.