Rely on aws-sdk

[rclark]

Full-blown refactor with some of the ideas laid out in #105. Still needs:

• discussion
• readme
• cli tests
• a pass to improve test coverage

cc @mick @willwhite

[rclark]

Here is some detail on the changes this PR would inflict. The overarching goal is to prefer to utilize the aws-sdk commands where possible, inheriting their bug fixes, updates, and documentation. Some aspects of dyno's API are really nice (e.g. .getItem(key) instead of .getItem({ Key: key })), but having documentation that we don't have to maintain is arguably better.

generally speaking

... the big changes are:

• "dyno-style" function arguments, conditions, and options go away. Instead all function arguments are as described for the aws-sdk's document client.
• dyno navigates the differences between the aws-sdk's regular client and document client for you behind the scenes.
• with the exception of explicit stream functions, none of the function calls fan out into multiple HTTP requests. It doesn't handle pagination, it doesn't split large batch requests into a series of acceptably-sized requests, and it doesn't attempt to retry errors. Configuring when and how errors are retried becomes the client's responsibility using aws-sdk's mechanisms.

breaking changes

• client configuration requires you to provide a table name
• dyno.getItems renamed to dyno.batchGetItem, as a passthrough to aws-sdk.
  • no longer accepts an array of keys and optional options object. See aws-sdk docs for parameters
  • does not allow you to request more than 100 items per function call
  • does not provide a readable stream of response records
• dyno.deleteItems and dyno.putItems replaced by dyno.batchWriteItem, a passthrough to aws-sdk.
  • no longer accepts an array of items to put / keys to delete and optional options object. See aws-sdk docs for parameters
  • does not allow you to write > 25 items or 16 MB per function call
• dyno.deleteItem becomes a passthrough to aws-sdk. No longer accepts a key and optional options object. See aws-sdk docs for parameters
• dyno.getItem becomes a passthrough to aws-sdk. No longer accepts a key and optional options object. See aws-sdk docs for parameters
• dyno.putItem becomes a passthrough to aws-sdk. No longer accepts a record and optional options object. See aws-sdk docs for parameters
• dyno.updateItem becomes a passthrough to aws-sdk. No longer accepts a key, "dyno-style" update definition and optional options object. See aws-sdk docs for parameters
• dyno.query becomes a passthrough to aws-sdk.
  • no longer accepts the "dyno-style" query conditions and optional options object. See aws-sdk docs for parameters
  • does not paginate through the reponse for you. Pagination can be accomplished manually or via aws-sdk's functions.
  • does not provide a readable stream of response records
• dyno.scan becomes a passthrough to aws-sdk.
  • no longer accepts an optional options object. [aws-sdk docs for parameters](See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#scan-property)
  • does not paginate through the response for you. Pagination can be accomplished manually or via aws-sdk's functions
  • does not provide a readable stream of response records
• drops dyno.estimateSize
• drops all kinesis-related functionality

new functions

• dyno.queryStream provides a readable stream of query responses. Parameters passed to the function are identical to dyno.query, but the function returns a readable stream that will paginate through results as you read from it.
• dyno.scanStream provides a readable stream of scan responses. Parameters passed to the function are identical to dyno.scan, but the function returns a readable stream that will paginate through results as you read from it.
• dyno.batchWriteItemRequests provides an array of AWS.Request objects representing BatchWriteItem requests. Parameters passed to the function are identical to dyno.batchWriteItem, except that there is no size/count limit to the number or write requests you may provide.
• dyno.batchGetItemRequests provides an array of AWS.Request objects representing BatchGetItem requests. Parameters passed to the function are identical to dyno.batchGetItem, except that there is no size/count limit to the number or get requests you may provide.

cc @mick @willwhite @miccolis

[willwhite]

Let's merge this. I'm fully onboard with the goal of maintaining less code and documentation (especially because the dyno docs to date have been... lacking). Looking back at the list of things dyno is really bringing to the table and it seems like the biggest ones are streaming support, multi-region support, and cli.

dyno navigates the differences between the aws-sdk's regular client and document client for you behind the scenes.

What does this mean? Just that you don't need to decide which client to use it advance and that dyno picks which client to use based on what calls you make?

with the exception of explicit stream functions, none of the function calls fan out into multiple HTTP requests

:clap:

As for when to bite off the upgrades for key downstream projects, we can play that by ear. I don't think it will be that bad if we split up the work.

[rclark]

By "navigates the clients" I mean dyno offers you one set of commands instead of the two that aws-sdk sometimes provides. For example:

• the regular client gives you a getItem function, the document client gives you a get function, dyno just has a getItem, but it relies on the document client
• the regular client has a createTable function but the document client does not, so dyno exposes the regular client's function
• the regular client and the document client both have scan functions, dyno chooses to expose the document client's function

[willwhite]

Gotcha, makes sense. Anything else before mergetown?

[rclark]

Yeah, the CLI needs to be updated/tested and there are some significant gaps in test coverage in the streaming code.

Also, while I agree that it is nice to know that dyno won't fan out your batch requests into multiple HTTP requests, I see us writing the same code in several places to take a list of things to get or things to write, and having to break that up into requestable chunks.

To solve this, how about a dyno function like:

dyno.batchWriteRequests = function(params) { ... };

... where params is identical to batchWriteItem parameters, but without limits on size/count. The function would return an array of AWS.Request objects that the caller could .send() in whatever concurrency settings they choose.

[willwhite]

@rclark adjustments look good to me. I'm on the fence about if a beta release is needed. If we do a 1.0.0 we can always do a 1.2.0 :)

[mick]

@rclark yeah should have released those were generated. There are some other formatting changes that happened when I ran npm run docs maybe changed in documentionjs? Or am I missing some other doc generation config?

otherwise LGTM

[rclark]

@mick note that package.json uses documentationjs@master. Maybe reinstall it and try the npm script again?

[mick]

@rclark hmm I just installed it for the first time. Maybe it changed since that last time it was run?

[mick]

I dont think it's an issue. I just wanted to flag for you since I wasnt familiar with that step in case I missed something.

[rclark]

No worries @mick. Leave it as is and we'll be in better shape once there's a new documentationjs tag.