Closed chadlagore closed 4 years ago
@chadlagore - thanks for bringing up the issue
list-shards will be a good alternative to describeShard. Would be great if you can open a PR for it.
In the absence of list-shards API, I can think of following ways (most of them you have already tried)
describeShardInterval
. 3600s (1 hr) would be a good value if you don't reshard too frequently.describeShardInterval
to different streaming queries. retryIntervalMs
say 2 secs. And have more retries - kinesis.client.numRetries
- say 10 or something like that.Thanks for the update. We've been able to increase this limit via AWS support for the time being - only to 20, but it gives us a bit of breathing room. We're migrated some of our other services to list-shards. I suspect we'll hit the limit again in the future, prior to which we'll probably have to make a change here.
Fix underway in https://github.com/qubole/kinesis-sql/issues/83
Currently seeing rate limit exceptions for DescribeStreams API. The reason is that we have 15-20 queries running in one application, and the DescribeStreams API only accepts 10 QPS per account. The trigger time is 30s, so what I believe is happening is the following:
describeShardInterval
has passed, and issues one API call per running query.retryIntervalMs
later, the remaining calls get issued, some more succeed, and so on.numRetries
is exhausted, the remainder fail.Options we've explored:
describeShardInterval
; did not work for obvious reasons, the thundering heard described above still takes place eventually.getLatestShardInfo
calls altogether; not actually ideal or possible, but we could do something like settingdescribeShardInterval
to very large number to accomplish in the short term. Getting new shards would then require a redeploy.retryIntervalMs
to >1s? The hypothesis being that this is small enough to cause some edge condition.Have you experienced this? I don't suppose it is possible to jitter the
describeShardInterval
per query? Is there another solution you could recommend? Happy to open a PR if it requires a change.I believe this is unrelated to https://github.com/qubole/kinesis-sql/issues/50.
EDIT: Also now noticing that list-shards might now be more appropriate to use?