qubole / kinesis-sql

Kinesis Connector for Structured Streaming
http://www.qubole.com
Apache License 2.0
137 stars 80 forks source link

LimitExceededException with multiple running queries per application #83

Closed chadlagore closed 4 years ago

chadlagore commented 4 years ago

Currently seeing rate limit exceptions for DescribeStreams API. The reason is that we have 15-20 queries running in one application, and the DescribeStreams API only accepts 10 QPS per account. The trigger time is 30s, so what I believe is happening is the following:

Options we've explored:

  1. Increasing describeShardInterval; did not work for obvious reasons, the thundering heard described above still takes place eventually.
  2. Adding some jitter to the trigger intervals. I don't see a spark config for this, but could probably fake it on a per query basis.
  3. Disabling getLatestShardInfo calls altogether; not actually ideal or possible, but we could do something like setting describeShardInterval to very large number to accomplish in the short term. Getting new shards would then require a redeploy.
  4. Increase retryIntervalMs to >1s? The hypothesis being that this is small enough to cause some edge condition.
  5. Requesting an increase in QPS on DescribeStreams (request submitted, unsure whether this will work)

Have you experienced this? I don't suppose it is possible to jitter the describeShardInterval per query? Is there another solution you could recommend? Happy to open a PR if it requires a change.

I believe this is unrelated to https://github.com/qubole/kinesis-sql/issues/50.

EDIT: Also now noticing that list-shards might now be more appropriate to use?

itsvikramagr commented 4 years ago

@chadlagore - thanks for bringing up the issue

list-shards will be a good alternative to describeShard. Would be great if you can open a PR for it.

In the absence of list-shards API, I can think of following ways (most of them you have already tried)

  1. Know your sharding frequency. Accordingly, use describeShardInterval. 3600s (1 hr) would be a good value if you don't reshard too frequently.
  2. Try to give different values of describeShardInterval to different streaming queries.
  3. yeah, we can start with a higher value of retryIntervalMs say 2 secs. And have more retries - kinesis.client.numRetries - say 10 or something like that.
chadlagore commented 4 years ago

Thanks for the update. We've been able to increase this limit via AWS support for the time being - only to 20, but it gives us a bit of breathing room. We're migrated some of our other services to list-shards. I suspect we'll hit the limit again in the future, prior to which we'll probably have to make a change here.

chadlagore commented 4 years ago

Fix underway in https://github.com/qubole/kinesis-sql/issues/83