DynamoDB documentClient support

fboudra commented 4 years ago

Is your feature request related to a problem? Please describe. Migrate existing code using documentClient to v3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html

I'm also using dynamodb-toolbox on some projects (it relies on documentClient ): https://github.com/jeremydaly/dynamodb-toolbox

Describe the solution you'd like ability to use the convenient documentClient API with v3

Describe alternatives you've considered

go back to primitive calls like client putitem/getitem
stick to v2 and don't migrate

russell-dot-js commented 4 years ago

I'm with you @fboudra, looking forward to DocumentClient! However, this is a duplicate of https://github.com/aws/aws-sdk-js-v3/issues/288

Maybe you or I could take a stab at porting over DocumentClient 🤔

mhart commented 4 years ago

It's a duplicate, but the other issue was closed so I'm assuming this is the best place to follow progress of this?

@AllanFly120 is there somewhere else tracking this?

antstanley commented 4 years ago

Hey folks, for those struggling with this I've created an npm package to convert JSON to DynamoDB AttributeValue type schema JSON and back again.

I'm using it today in a small production project.

I've created two packages, one a CommonJS module, and the other a ES module. You can find them, with docs here.

It's a fork and update from the DocumentClient converter.js code.

CommonJS module - dynamodb-type-marshall https://www.npmjs.com/package/dynamodb-type-marshall

ES module - @esmodule/dynamodb-type-marshall https://www.npmjs.com/package/@esmodule/dynamodb-type-marshall

Source code here: https://github.com/senzodev/dynamodb-type-marshall

trivikr commented 4 years ago

Thanks @antstanley for creating a package for marshalling and unmarshalling.

We've implemented marshaller (convertToAttr) in https://github.com/aws/aws-sdk-js-v3/pull/1531, and aiming to push it in gamma.10 launch before 9/25 A PR for unmarshaller (convertToNative) will be posted soon.

trivikr commented 4 years ago

The support for marshall and unmarshall operations was released in gamma.10 release, and verified with the code below:

Versions tested on:

    "@aws-sdk/client-dynamodb": "1.0.0-gamma.10",
    "@aws-sdk/util-dynamodb": "1.0.0-gamma.1"

Code

```js const { DynamoDB } = require("@aws-sdk/client-dynamodb"); const { marshall, unmarshall } = require("@aws-sdk/util-dynamodb"); const { deepStrictEqual } = require("assert"); (async () => { const region = "us-west-2"; // Table with Partition key named `HashKey` const TableName = `test-util-dynamodb`; const HashKey = "hashKey"; const input = { HashKey, BinaryAttribute: Uint8Array.from("12345"), BoolAttribute: true, BoolSetAttribute: new Set([Uint8Array.from("1"), Uint8Array.from("2")]), ListAttribute: [1, "two", false], MapAttribute: { foo: "bar" }, NumAttribute: 1, NumSetAttribute: new Set([1, 2, 3, 4, 5]), NullAttribute: null, StrAttribute: "one", StrSetAttribute: new Set(["one", "two", "three"]), }; const Item = marshall(input); console.log({ putItem: input }); const client = new DynamoDB({ region, // logger: { ...console, debug: undefined }, }); await client.putItem({ TableName, Item }); const response = await client.getItem({ TableName, Key: marshall({ HashKey }), }); console.log({ getItem: unmarshall(response.Item) }); deepStrictEqual(input, unmarshall(response.Item)); })(); ```

Output

```console { putItem: { HashKey: 'hashKey', BinaryAttribute: Uint8Array(5) [ 1, 2, 3, 4, 5 ], BoolAttribute: true, BoolSetAttribute: Set { [Uint8Array], [Uint8Array] }, ListAttribute: [ 1, 'two', false ], MapAttribute: { foo: 'bar' }, NumAttribute: 1, NumSetAttribute: Set { 1, 2, 3, 4, 5 }, NullAttribute: null, StrAttribute: 'one', StrSetAttribute: Set { 'one', 'two', 'three' } } } { getItem: { ListAttribute: [ 1, 'two', false ], MapAttribute: { foo: 'bar' }, BinaryAttribute: Uint8Array(5) [ 1, 2, 3, 4, 5 ], StrAttribute: 'one', BoolSetAttribute: Set { [Uint8Array], [Uint8Array] }, NumSetAttribute: Set { 5, 4, 3, 2, 1 }, BoolAttribute: true, StrSetAttribute: Set { 'one', 'three', 'two' }, HashKey: 'hashKey', NumAttribute: 1, NullAttribute: null } } ```

If you come across any bugs, do report them by creating an issue: https://github.com/aws/aws-sdk-js-v3/issues/new?assignees=&labels=bug+report&template=---bug-report.md&title=

antstanley commented 4 years ago

Awesome! Will give it a go today.

trivikr commented 4 years ago

Closing as the core operations, i.e. marshall/unmarshall, were released in gamma.10 release. Currently we are not planning to provide support for a separate DocumentClient.

mhart commented 4 years ago

@trivikr will you be providing guidance on how to migrate from DocumentClient? Would be good to see how easy it is to go from DocumentClient to marshall/unmarshall

trivikr commented 4 years ago

The utility operations marshall and unmarshall are exposed in @aws-sdk/util-dynamodb, and here’s a code showing how to use them:

const { DynamoDB } = require("@aws-sdk/client-dynamodb");
const { marshall } = require("@aws-sdk/util-dynamodb");

const client = new DynamoDB(clientParams);
const putParams = {
  TableName: "Table",
  Item: marshall({
    HashKey: "hashKey",
    NumAttribute: 1,
    BoolAttribute: true,
    ListAttribute: [1, "two", false],
    MapAttribute: { foo: "bar" },
    NullAttribute: null,
  }),
};

await client.putItem(putParams);

const getParams = {
  TableName: "Table",
  Key: marshall({
    HashKey: "hashKey",
  }),
};

const { Item } = await client.getItem(getParams);
unmarshall(Item);

The examples are also provided in README at: https://github.com/aws/aws-sdk-js-v3/tree/v1.0.0-rc.3/packages/util-dynamodb We're working on updating the Developer Guide with this example.

stang-tgs commented 3 years ago

Does it not make sense for a DynamoDB JavaScript library to provide the DocumentClient functionality? I mean, under what circumstances in JavaScript/TypeScript would NOT use native JS objects and would want to use the marshalled DynamoDB types? I'd hazard a guess to say almost never. So, not providing a DocumentClient seems to be a straight annoyance for most SDK users, because, chances are, marshal() and unmarshall() is going to be sprinkled before/after almost every database operation where data types need to be specified/used - I can't imagine why a JS/TS developer would ever need to operate on a native, marshalled DynamoDB data model in JS.

I've started to port our existing TS code from aws sdk v2 to v3 where we use DocumentClient. Fortunately, we have an abstraction layer, so we are able to marshall/unmarshall there without affecting calling code. Unfortunately, we have to intercept and manually sprinkle in marshall/unmarshall everywhere now, which I suspect many developers porting would have to do as well.

I'm still going through the motions of doing the marshalling/unmarshalling - but my 2 cents is that it's an oversight and disservice by not providing a DocumentClient. If the .NET/Java library required developers to manually marshall/unmarshall, I'm quite sure everyone's heads will explode 😉.

trivikr commented 3 years ago

Reopening the issue to explore addition of DocumentClient.

reni11111 commented 3 years ago

+1 Saying you don't need DocumentClient because u can marshall and unmarshall by yourself is the same as saying you don't need .map .reduce .filter .find because you can do a for loop and do it manually.

eryon commented 3 years ago

I don't really know how safe this is, but I'm now using this to auto-marshall:

  client.middlewareStack.add(
    (next) => async (args) =>
      next({
        ...args,
        input: Object.entries(args.input).reduce(
          (acc, [key, value]) => ({
            ...acc,
            [key]: ['ExpressionAttributeValues', 'Key', 'Item'].includes(key)
              ? marshall(value, { convertEmptyValues: true })
              : value
          }),
          {}
        )
      }),
    {
      name: 'autoMarshall',
      priority: 'high',
      step: 'initialize'
    }
  );
  client.middlewareStack.add(
    (next) => async (args) => {
      const result = await next(args);

      return {
        ...result,
        output: {
          ...result.output,
          Item: result.output.Item && unmarshall(result.output.Item),
          Items: result.output.Items && result.output.Items.map((i) => unmarshall(i))
        }
      };
    },
    {
      name: 'autoUnmarshall',
      priority: 'high',
      step: 'deserialize'
    }
  );

where client is a new DynamoDBClient and this is in a common utility function.

mattiLeBlanc commented 3 years ago

Glad to have the marshall/unmarshall option. However, the V2 DocumentClient was easier to work with. So I would also love to see it migrated to V3.

ghost commented 3 years ago

~~AWS SDK version 3 will be supported soon in this package. https://gitlab.com/monster-space-network/typemon/dynamon~~

ghost commented 3 years ago

I recommend the @typemon/dynamon package. AWS SDK version 3 is supported, marshall and unmarshall are handled automatically. It also provides the ability to write expressions in a very easy way.

yaquawa commented 3 years ago

any updates on this??

trivikr commented 3 years ago

Updates in @aws-sdk/util-dynamodb:

v3.4.0: added option convertClassInstanceToMap to convert class instance to map https://github.com/aws/aws-sdk-js-v3/pull/1969
v3.5.0 (expected release on 02/11): support marshalling for Object.create() in https://github.com/aws/aws-sdk-js-v3/pull/1974

We're not planning to support a separate DocumentClient in v3 as of now. Are there any specific features you're missing in @aws-sdk/util-dynamodb? Please create an issue to submit a request.

mattiLeBlanc commented 3 years ago

@trivikr I added one suggestion for the util-dynamodb here: https://github.com/aws/aws-sdk-js-v3/issues/2009

cadam11 commented 3 years ago

@trivikr The downside of having to manually marshall/unmarshall is that the user now has to keep track of which params need to be marshalled/unmarshalled. IMO, that's too much internal architecture knowledge to require from consumers.

If there's to be no DocumentClient in v3, how about adding a first-party middleware for auto-marshall/unmarshall similar to what @eryon proposed? @aws-sdk/middleware-automarshall or something?

trivikr commented 3 years ago

If there's to be no DocumentClient in v3, how about adding a first-party middleware for auto-marshall/unmarshall similar to what @eryon proposed? @aws-sdk/middleware-automarshall or something?

A separate DocumentClient was not written for v3 mainly because of the difference in types: AttributeValue from DynamoDB, and NativeAttributeValue for native JavaScript types. We will have to emit redundant models for DynamoDB operations which use AttributeValue or a separate client, and this work would have to be codegen as DynamoDB service may decide to introduce new operations which use AttributeValue.

We also evaluated automatically detecting whether customer code has sent an attribute value or native JavaScript types based on the actual data. But it had multiple edge cases. Another option we evaluated is allowing customer to configure whether DynamoDB records or native JavaScript values are passed in an operation. But that option also had some edge cases, and added overhead for customer to understand the configuration. I can provide specific details if required.

After a week long design discussion, we decided that the utility functions in @aws-sdk/util-dynamodb would be the best solution. It would be difficult to switch to in the short-term, but would be beneficial in the long term. The developer resources can be diverted for designing a higher level library like DynamoDB Datamapper instead.

@cadam11 Do you have a suggestion for a middleware @aws-sdk/middleware-automarshall which can work without changing DynamoDB input/output type definitions?

cadam11 commented 3 years ago

I guess it depends on how strict you are about

without changing DynamoDB input/output type definitions

At the very least, I'd expect that for a middleware to work there would need to be some (non-breaking) additions to the existing types. E.g., Item would have to allow an alternative to a map of AttributeValues– I'm imagining some use of generics to provide a return type for GetItem, say.

Wish I had more time to dig into this interesting problem. In any case, I'm still a fan of v3 sdk.

kyeotic commented 3 years ago

@trivikr That doesn't make a whole lot of sense. Your first paragraph lays out the difficulty in doing this work, and your penultimate paragraph explains how you decided that its better that every consumer of your library do that work instead.

That's what a library is for: to do the difficult work that everyone needs once. You recognize that it's awkward for the consumer of the unmarshalled data to work with the underlying types. That's the reason that you should do it!

trivikr commented 3 years ago

The first paragraph explains difficulty in doing this work when it comes to being an SDK owner. This involves writing and maintaining a separate DocumentClient and emitting mostly redundant type definitions.

The penultimate paragraph explains how easy it for customer to just use marshall/unmarshall functions which takes native JavaScript types as inputs, and return DynamoDB records as output. This way:

Customer can pass continue passing JavaScript types which they used for previous DocumentClient.
Customers don't have to make additional choice between DynamoDB client and DocumentClient.

Can you create a separate feature request on what specific difficulties you're facing when moving from DynamoDB DocumentClient in v2 to utilities in v3?

I'm writing a blog post on DynamoDB utilities, and aiming to publish in a month. I'll include some scripts to ease migration based on the problems faced.

kyeotic commented 3 years ago

The difficulty is writing the same marhsall/unmarshall code and maintaining those "mostly redundant types" either in every app that uses the aws sdk or in our own library that provides the same.

This is a JavaScript library that doesn't work with JavaScript objects! What JavaScript app wants to interact with those DynamoDB marshalled objects? I'm guessing next to none. Why not abstract away the transport format in the library that interacts with the transport layer?!

fboudra commented 3 years ago

@trivikr there's no point to create another feature request. It's exactly what this issue is about. It's already stated in my initial feature request at the top of this thread: Migrate existing code using documentClient to v3

The fact is that's a pain to migrate to your recommended marshall/unmarshall approach.

Feel free to read https://github.com/jeremydaly/dynamodb-toolbox/issues/52 and see how it creates a massive breaking change for developers consuming the SDK. Isn't it ironic for a SDK?

jamesrenaud commented 3 years ago

Feel free to read jeremydaly/dynamodb-toolbox#52 and see how it creates a massive breaking change for developers consuming the SDK. Isn't it ironic for a SDK?

I mean, this is a v2 to v3 major version bump, breaking changes are to be expected.

trivikr commented 3 years ago

I mean, this is a v2 to v3 major version bump, breaking changes are to be expected.

Yup, the modular AWS SDK for JavaScript (v3) comes with modular architecture which significantly reduces application bundle size, adds first-class TypeScript support for customers using TypeScript and a new middleware stack for faster debugging.

Providing DynamoDB utility functions instead of DocumentClient is another decision we had to take because of reasons explained in https://github.com/aws/aws-sdk-js-v3/issues/1223#issuecomment-776285213

Having said that, we would still like to help migrate customers from DocumentClient to DynamoDB utilities.

Can you create a separate feature request on what specific difficulties you're facing when moving from DynamoDB DocumentClient in v2 to utilities in v3?

I'm writing a blog post on DynamoDB utilities, and aiming to publish in a month. I'll include some scripts to ease migration based on the problems faced.

stang-tgs commented 3 years ago

Just my 2 cents here. Having provided marshall/unmarshall in @aws-sdk/util-dynamodb, I was able to migrate (our TypeScript projects), with minor pain, to v3 from v2 DocumentClient. Luckily for my projects, we have an abstraction layer where we were able to do the marshalling/unmarshalling in a single place, before sending the DynamoDB command. This may not be true for everyone though.

The initial versions of marshall/unmarshall were not directly functioning as v2 DocumentClient due to narrowed marshalling rules, but seems to be more compatible now with options in #1969, though I haven't tested yet. Our TypeScript projects have been using v3 for more than a month since our port (with some hacks, prior to #1969), and for the most part, functional.

I agree that AWS DynamoDB SDK should be as simple as possible and provide the lego blocks for developers to do as they will.

A separate DocumentClient was not written for v3 mainly because of the difference in types: AttributeValue from DynamoDB, and NativeAttributeValue for native JavaScript types. We will have to emit redundant models for DynamoDB operations which use AttributeValue or a separate client, and this work would have to be codegen as DynamoDB service may decide to introduce new operations which use AttributeValue.

Ironically, both the C# and JAVA AWS SDK's, both type strict languages, provide some sort of mapper and document model via first party support from Amazon, and probably have similar challenges. Although TypeScript is also type strict and JavaScript being less so, one would think that it should be the same effort, if not easier for JavaScript than for C# and JAVA. I wonder if C# and Java will drop their mapper support at some point too then, due to the effort❓

The debate is on who provides the marshalling/mapper service, Amazon, or 3rd party, and to what extent (e.g. simple enough, but not as complicated as a full blown ORM - we'll leave the full ORM to 3rd parties and to people who want them, regardless of whether ORMs should be used for DynamoDB or not😉)

Despite the criticism, I want to thank the AWS SDK developers for v3 - it's much cleaner, more modular, plus the better support (stack traces) for async/await/promises.

mdesousa commented 3 years ago

@trivikr , if I understand your comment about the DynamoDB Datamapper... it sounds like this is going to be the recommended approach for JavaScript that will support native types and eliminate the need to marshall/unmarshall. So in a sense, the Datamapper would be an improvement over DocumentClient. Am I understanding that correctly? If that is the case, this sounds promising to me... I'm familiar with the existing data mappers for .NET and Java and those are pretty easy to use. Does the Javascript datamapper already support the aws sdk v3?

cadam11 commented 3 years ago

better support (stack traces) for async/await/promises.

Worth the price of admission right there. I can't count how many times I've facepalmed for forgetting .promise(). I agree it would be nice to have better abstraction from DynamoDB attributes, but that doesn't diminish the goodness of v3 for me.

kyeotic commented 3 years ago

@trivikr Ive spent the last two hours trying to write a sane marshall/unmarshall for batchWrite. This is the best I can come up with.

async batchGet(params: BatchGetItemInput): Promise<BatchGetItemOutput> {
    if (params?.RequestItems) {
      params = {
        ...params,
        RequestItems: Object.fromEntries(
          Object.entries(params?.RequestItems).map(([table, attrs]) => [
            table,
            {
              ...attrs,
              Keys: attrs.Keys?.map(this.marshall),
            },
          ])
        ),
      }
    }
    const response = await this._dynamo.send(new BatchGetItemCommand(params))
    response.Responses =
      response.Responses &&
      Object.fromEntries(
        Object.entries(response.Responses).map(([table, items]) => [
          table,
          items.map(this.unmarshall),
        ])
      )
    response.UnprocessedKeys =
      response.UnprocessedKeys &&
      Object.fromEntries(
        Object.entries(response.UnprocessedKeys).map(([table, attrs]) => [
          table,
          {
            ...attrs,
            Keys: attrs.Keys?.map(this.marshall),
          },
        ])
      )
    return response
  }

These are the hoops everyone who uses the SDK has to jump through in order to use JavaScript objects with this JavaScript library

Edit: I wonder if anyone noticed the marshalling error in the code above. It took me a bit, too. The UnprocessedKeys actually need to be unmarshalled... of course they will need to be re-marshalled on the next go... and then unmarshalled again... and so on, etc.

trivikr commented 3 years ago

Does the Javascript datamapper already support the aws sdk v3?

Not yet. The existing @aws/dynamodb-data-mapper was an experimental package built on top of JS SDK v2.

There is already an existing open issue requesting for DynamoDB data mapper in v3 at https://github.com/aws/aws-sdk-js-v3/issues/1085 Considering it's popularity in terms of reactions, and the comments on DocumentClient support - we should consider prioritizing it. I'll raise this request in our planning meeting.

trivikr commented 3 years ago

Ive spent the last two hours trying to write a sane marshall/unmarshall for batchWrite. This is the best I can come up with.

@kyeotic Can you create a feature request for this specific issue? We could develop something similar to https://github.com/aws/aws-sdk-js-v3/issues/2009 for batchGet or similar operations where usage of marshall/unmarshall is not straightforward.

kyeotic commented 3 years ago

@trivikr This issue is that feature request

trivikr commented 3 years ago

This issue is that feature request

This issue is a generic feature request for DocumentClient support, and it got lot of comments. We worked on specific implementations for example:

support to convert class instance to map https://github.com/aws/aws-sdk-js-v3/pull/1969
support to marshall objects created with Object.create https://github.com/aws/aws-sdk-js-v3/pull/1974
support for marshalling JavaScript Maps https://github.com/aws/aws-sdk-js-v3/pull/2010

A specific feature request would help providing support for those issue. Let me create one for batchGet.

kyeotic commented 3 years ago

@trivikr I don't want a solution just for batchGet. I want a solution for all the methods that require marshalling; I want the equivalent of the v2 DocumentClient.

I want this JavaScript library to work with JavaScript objects in general, not just in specific cases. Making it work only in specific cases might actually be worse, since developers would then need to understand which cases worked and which didn't.

I'm sorry if focusing on batchGet was confusing, I posted that example because it took me a while to get it working with typescript. Once I finish all of the data-writing methods I'm happy to post them all here if that helps keep the issue focused on the big picture.

kyeotic commented 3 years ago

It's already been said a few times in this thread, but I'm going to take one more stab.

Everyone working in JavaScript who uses this library wants to use JavaScript types, not DynamoDB types. Everyone.

I know this because nothing else uses DynamoDB's types, so doing any work at all with the output from this library means unmarshalling the data. Let me repeat that: using any data that comes out of this library requires that it be processed by another library before it can be used by normal JavaScript code. Likewise, anyone using data from outside needs to marshall the data first because nothing outside this library used DynamoDB types, it all uses JavaScript objects. Before providing any data to this library it first requires that it be processed by another library.

You have to see how painful and un-ergonomic that is. The purpose of a JavaScript DynamoDB library should be to translate from Dynamo to JavaScript. This library only provides half an abstraction.

Giving and requiring DynamoDB data was the wrong call. If you don't want to maintain both types then get rid of the Dyanmo types. This library is the only one that should be working with them anyway.

The original request has 40 upvotes. This explanation higher in the thread saying what I'm saying here has 30.

You guys are going in the wrong direction.

cadam11 commented 3 years ago

Everyone working in JavaScript who uses this library wants to use JavaScript types, not DynamoDB types. Everyone.

Gotta echo this-- I have never had a use for DynamoDB types.

kyeotic commented 3 years ago

For everyone else that needs Typescript types for native (marshalled/unmarshalled) client methods I made a gist.

trivikr commented 3 years ago

A draft PR is posted with DocumentClient put operation at https://github.com/aws/aws-sdk-js-v3/pull/2031

Do comment on it, while I work on polished PR on a different branch. Tagging @kyeotic @cadam11 @mdesousa @stang-tgs @fboudra @mattiLeBlanc

rati-dzidziguri commented 3 years ago

While I understand the request to have DocumentClient, I have to admit that modular architecture is beneficial. Smaller bundle size is significant when it comes down to Serverless and Lambda. We saw significant improvement in both cold start time and overall memory allocation.

https://github.com/jeremydaly/dynamodb-toolbox/issues/52 is one of the best libraries that exist out there ATM; however, the missing capability of Document Client blocks the owner to port it to the latest SDK version. @trivikr it would be awesome if you could help the library mentioned above to migrate to V3

mobsense commented 3 years ago

We've got the same issue as toolbox in our DynamoDB OneTable library: https://github.com/sensedeep/dynamodb-onetable.

We accept a DocumentClient instance into our OneTable constructor and use it for converting from Javascript objects to DynamoDB types. It would be really helpful if V3 could support DocumentClient capabilities and/or provide an upgrade path for DocumentClient apps and libraries. If not, we will need to include the marshaling to and from Javascript objects regardless, which sort of defeats the smaller size argument as you end up with that code being included in either case.

trivikr commented 3 years ago

For modular implementation, the AWS SDK for JavaScript team is exploring two different solutions.

Command code:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
// import to make the call.

const client = new DynamoDBClient({});
const getParams = {
  TableName: 'TABLE_NAME',
  // as opposed to { 'KEY_NAME': { 'ATTRIBUTE_VALUE': VALUE } }
  Key: { 'KEY_NAME': VALUE }
};
const marshallOptions = {}; // options to use while marshalling.
const unmarshallOptions = {}; // options to use while unmarshalling.

// Code to make the call.

// Item is of type { 'KEY_NAME': VALUE }
const { Item } = response;

Option ❤️ is similar to any other Command from DynamoDBClient:

// import to make the call.
import { GetNativeItemCommand } from "@aws-sdk/lib-dynamodb";

// Code to make the call.
const response = await client.send(
  new GetNativeItemCommand(getParams, { marshallOptions, unmarshallOptions });
);

Option 🎉 is similar to Upload operation in @aws-sdk/lib-storage:

// import to make the call.
import { Get } from "@aws-sdk/lib-dynamodb";

// Code to make the call.
const response = Get({client, {marshallOptions, unmarshallOptions}}, getParams);

Do respond on this issue comment with reaction ❤️ or 🎉 to specify your preference.

Tagging @kyeotic @cadam11 @mdesousa @stang-tgs @fboudra @mattiLeBlanc @rati-dzidziguri @mobsense @russell-dot-js @mhart @antstanley

kyeotic commented 3 years ago

@trivikr here is how I implemented marshalling all the document client calls (or here to see query specifically) while retaining the modular/tree-shakable input from the sdk.

Its closer to the ❤️ option, but it keeps the marshall/unmarshall options tied to the client.

rati-dzidziguri commented 3 years ago

The option ❤️ - is similar to any other Command from DynamoDBClient: it seems more natural. If somebody tried to create a direct wrapper around this, I would prefer to see a minimum split around readers and writes as this makes a big diff. While some microservices are only for reading the data, others care about writing only. so splitting that wrapper around readers and writers would look natural as well

//Import reader client
import { DynamoDBReaderClient } from "@aws-sdk/client-dynamodb"

//Import writer client
import { DynamoDBWriterClient } from "@aws-sdk/client-dynamodb"

This way, we make sure that the package still does not include specific commands unless those are required.

mobsense commented 3 years ago

Just FYI: Previously I said it would be really helpful to have DocumentClient supported for use in our OneTable library. However, after digging deeper, we were able to support the V3 SDK pretty easily without this and we just released an update of OneTable with V3 support.

The marshall / unmarshall library did most of the work and we were able to maintain 100% API compatibility for OneTable users after initialization of the library. Users have 2 code lines of difference in using V2 or V3 of the SDK. All smiles here.

So while DocumentClient support doesn't matter for us now, it do believe that for many developers, having a compatibility shim so they don't need to rewrite their DynamoDB apps that use DocumentClient would be worthwhile.

G-Rath commented 3 years ago

If it helps, here's a type-based Marshall:

import { AttributeValue, DynamoDB } from '@aws-sdk/client-dynamodb';

type GenericSMember<S extends string> = AttributeValue.SMember & { S: S };

type MarshallProperty<T> = T extends string
  ? GenericSMember<T>
  : T extends number
  ? AttributeValue.NMember
  : T extends boolean
  ? AttributeValue.BOOLMember
  : T extends Uint8Array
  ? AttributeValue.BMember
  : T extends Set<string>
  ? AttributeValue.SSMember
  : T extends Set<number>
  ? AttributeValue.NSMember
  : T extends Set<Uint8Array>
  ? AttributeValue.BSMember
  : T extends null
  ? AttributeValue.NULLMember
  : T extends Array<infer TItems>
  ? MarshallProperty<TItems>
  : T extends Record<infer TKeys, unknown>
  ? MarshallProperty<TKeys>
  : never;

type Marshall<T extends object> = { [K in keyof T]: MarshallProperty<T[K]> };

Example usage:

interface PullRequestOutcome {
  pk: 'PullRequest';
  sk: `${string}${string}#${string}${string}`;
  host: 'github' | 'bitbucket';
  conclusion: 'merged' | 'declined';
  openedAt: string;
  closedAt: string;
}

const Item: Marshall<PullRequestOutcome> = {
  pk: { S: '' },
  sk: { S: 'my-project#12' },
  host: { S: 'github' },
  conclusion: { S: 'declined' },
  openedAt: { S: '' },
  closedAt: { S: '' }
};

It should work fine for all things except for tuples (as they can't be preserved properly, so they'll just become Array).

I've also got an Unmarshall, but that's less accurate:

type UnmarshallProperty<T> = T extends GenericSMember<infer S>
  ? S
  : T extends AttributeValue.NMember
  ? number
  : T extends AttributeValue.BOOLMember
  ? boolean
  : T extends AttributeValue.BMember
  ? Uint8Array
  : T extends AttributeValue.SSMember
  ? Set<string>
  : T extends AttributeValue.NSMember
  ? Set<number>
  : T extends AttributeValue.NULLMember
  ? null
  : T extends Record<string, unknown>
  ? UnmarshallProperty<T>
  : never;

type Unmarshall<T extends object> = {
  [K in keyof T]: UnmarshallProperty<T[K]>;
};

declare const unmarshall: <T extends { [key: string]: AttributeValue }>(
  item: T
) => Unmarshall<T>;

const item = unmarshall({
  pk: { S: 'PullRequest' as const },
  sk: { S: 'my-project#12' },
  host: { S: 'github' },
  conclusion: { S: 'declined' },
  openedAt: { S: '' },
  closedAt: { S: '' },
});

trivikr commented 3 years ago

Update: After the design discussion within the team, a PoC for DocumentClient is posted at https://github.com/aws/aws-sdk-js-v3/pull/2062

This is a manually written PoC. In the final PR, the operations are going to be code generated and not manually written.

The modular DynamoDBDocumentClient can be created as follows:

  const client = new DynamoDBClient({});
  const ddbDocClient = DynamoDBDocumentClient.from(client);
  await ddbDocClient.send(
    new PutCommand({
      TableName,
      Item: { id: "1", content: "content from DynamoDBDocumentClient" },
    })
  );

The v2 backward-compatible DocumentClient can be created as follows:

  const client = new DynamoDBClient({});
  const ddbDoc = DynamoDBDocument.from(client);
  await ddbDoc.put({
    TableName,
    Item: { id: "2", content: "content from DynamoDBDocument" },
  });

trivikr commented 3 years ago

Update: The existing marshalling/unmarshalling code for BatchGet/BatchWrite is pretty complex. The code would be black box to users, but they may go through it while debugging.

You can view the current implementation for posting suggestions/comments at https://github.com/trivikr/aws-sdk-js-v3/blob/058c2078a078a8f1e3d27a34b10955f4a27752df/lib/lib-dynamodb/src/commands/BatchWriteCommand.ts

The PoC PR can be viewed at https://github.com/aws/aws-sdk-js-v3/pull/2062

Tagging @kyeotic as they had attempted writing marshall/unmarshall for batchWrite earlier in https://github.com/aws/aws-sdk-js-v3/issues/1223#issuecomment-777852837

aws / aws-sdk-js-v3

DynamoDB documentClient support #1223