search

mapbox / dyno

simple dynamodb client
MIT License
78 stars 31 forks source link

Implement backoff on batchWriteAll and batchGetAll #120

Closed mcwhittemore closed 7 years ago

mcwhittemore commented 8 years ago

@rclark Here's my first stab at adding backing off to batchWriteAll and batchGetAll.

I'm not sure I'm testing the right things here. I added one that times out for get and one that times out for write. I assumed these would give me a list of unprocessed items, but I guess they don't. Is it supposed to be that way?

rclark commented 8 years ago

The backoff is great, but the idea with these functions should be that you provide batch request parameters without limits on count or size, then dyno makes all the requests for you and returns a single response that looks just like what you'd have gotten from a single batch request.

So in the event that there are unprocessed items and you've hit the max retry count, we should return a response that looks like what you would've gotten from a single batch request with unprocessed items -- that is -- no error, and a response object with an .UnprocessedItems or .UnprocessedKeys property.

mcwhittemore commented 8 years ago

Ok. So even if it times out we don't treat it like an error. Will do.

mcwhittemore commented 8 years ago

@rclark - so, before this PR dyno was returning an unprocessed array, now its returning UnprocessedItems or UnprocessedKeys as part of the response object. That's what you were requesting above correct?

rclark commented 8 years ago

Before this PR, these functions would never return any unprocessed items. If there were any, it would keep retrying forever until there weren't... which was a terrible idea. Better is as you've done: configurable max retries, and then fail. What I'm asking is that when that happens, the unprocessed requests are returned as UnprocessedItems / UnprocessedKeys in the response object.

mcwhittemore commented 8 years ago

Ah. Sorry. sendAll returns an unprocessed array.

This is ready for another look when you get a chance. I'm going to start seeing how this works with cardboard.

The UnprocessedItems vs UnprocessedKeys shapes is a bit confusing in the Amazon docs but I think they are right.

mcwhittemore commented 7 years ago

Replaced by #122