Netflix / dyno

Java client for Dynomite
https://github.com/Netflix/dynomite
Apache License 2.0
186 stars 94 forks source link

ability to specify count hint for SCAN call #144

Open maksimlikharev opened 7 years ago

maksimlikharev commented 7 years ago

default redis count of elements for scan is 10, that results in long iteration cycle as it usually returns 1-2 elements per call. To optimize, function should accept count hint as well.

ipapapa commented 7 years ago

I am not following what is your question... You can use a cursor.

maksimlikharev commented 7 years ago

Redis scan command: SCAN cursor [MATCH pattern] [COUNT count]

"COUNT the user specified the amount of work that should be done at every call in order to retrieve elements from the collection."

by default COUNT is 10.

Specifying different counts I can influence # of records returned by scan

specifically in Jedis it is configured by ScanParams.count()

for dyno_scan, there is NO way to influence # of records returned by passing count hint to Redis.

Why this is important, I observe that dyne_scan returns at most 2 records per invocation, with 1000s of keys in my use-case I have 1000s of iteration over the cursor to retrieve them all.

With this said, additional parameter for count needs to be added to dyne_scan signature

maksimlikharev commented 7 years ago

also it is not clear to me why pattern is defined as vararg, assuming that Redis scan supports only one pattern, at least no indication in docs about alternative behavior.

ipapapa commented 7 years ago

By default dyno_scan returns 10, and if there are more than 10 keys, the client should return 10 every time (unless there are fewer keys). Hence most probably something is erroring out.

In regards to the COUNT, you are correct. It can definitely be added. It should probably be another ScatterGatherScan method with COUNT as argument.