search

qubole / kinesis-sql

Kinesis Connector for Structured Streaming
http://www.qubole.com
Apache License 2.0
137 stars 80 forks source link

Fix listShards to only specify nextToken #104

Open Anthuang opened 3 years ago

Anthuang commented 3 years ago

This fixes two bugs:

  1. If listShards returns multiple pages, then we run into an infinite loop with while (!nextToken.isEmpty), because nextToken will never be set back to null because: 
    if (returnedToken != null) {
nextToken = returnedToken
  2. listShards cannot allow both StreamName and NextToken to be specified together.
chadlagore commented 3 years ago

@itsvikramagr this is tested against streams with >1k shards which forces paging on the Kinesis API

itsmesrds commented 2 years ago

Does this read the equal number of messages too ? because I see there is a data skew.