search

aws / aws-sdk-ruby-record

Official repository for the aws-record gem, an abstraction for Amazon DynamoDB.
Apache License 2.0
318 stars 41 forks source link

Capacity used for iterating through a query's results #109

Closed linkyndy closed 4 years ago

linkyndy commented 4 years ago

I am putting together a script that crunches some numbers while iterating through the results of a query (made on an index). While running it, I can see that the read capacity consumed is very low (between 2-4). We have autoscaling in place for that table/index, and the current provisioned capacity is now at 8.

My question is, what stops aws-record to use the full capacity available on the index? And how can I make it use more?

mullermp commented 4 years ago

Hi @linkyndy, are you by chance configuring the table in code using Table Config and overriding auto scaling? Or perhaps have you set a low target utilization as described in this doc?

linkyndy commented 4 years ago

Hi @mullermp, thanks for your reply. After some digging I found that actually the script was fairly slow, and that was the actual capacity consumed, without further concurrency. Iterating through such large query results in a single thread takes time, especially with a lot of roundtrips to DynamoDB.