MIgration to Dynamodb V2 Is Really Rough

patrickjamesbarry commented 7 months ago

Describe the issue

We have hit one difference after another migrating to v2. Why is there nothing documented on regarding dynamodb differences between v1 and v2 on https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-service-changes.html ?

We have had to navigate huge differences between v1 (ie DynamodbMapper) to v2 EnhancedClient with only the information from StackOverflow and Github issues to help us understand. Examples of hurdles:

Default mode of consistent read in v1 is now something you have to opt into with v2
There is no supported "count" query in v2. Best option is the hack provided by user in https://github.com/aws/aws-sdk-java-v2/issues/2083
Fluent builder pattern on table beans/pojos does not work in v2. It did in v1 - https://github.com/aws/aws-sdk-java-v2/issues/5039

Every day we find some difference. This migration has NOT been easy and it is tied to 2 things. Lack of documentation and tickets/feature requests that have been logged years ago around feature parity with v1 and still have not been worked.

Links

https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-service-changes.html

debora-ito commented 7 months ago

@patrickjamesbarry thank you for raising this, and we hear you. We are actively working with the Documentation team to improve the Migration section in our Developer Guide, DynamoDB Enhanced is in the list but not being worked on yet.

In the meantime, I'll try to answer each question here:

Hurdle 1 - Default mode of consistent read in v1 is now something you have to opt into with v2

Are you talking about the v1 DynamoDBMapperConfig#withConsistentReads? The default ConsistentReads should be EVENTUAL in both v1 DynamoDBMapper and v2 DynamoDbEnhanced Client, can you share where you're seeing differently?

Hurdle 2 - There is no supported "count" query in v2

Actually, "count" was added via https://github.com/aws/aws-sdk-java-v2/pull/4444 in SDK version 2.20.153, I missed updating the tracking feature request. Note that query is a paginated operation, so "count" was added to each Page response.

Hurdle 3 - Fluent builder pattern on table beans/pojos does not work in v2

I don't think we had in scope to make the DynamoDb Enhanced Client work with fluent setters, but if it works in v1 we should support in v2. Will add the issue to our backlog.

Please keep posting questions regarding the migration to DynamoDB Enhanced client, we'll use them to improve our documentation. Other feedbacks are also welcome.

Lirons01 commented 7 months ago

@debora-ito Thank you for the response on this issue, the main difficulty that I've noticed in the migration from sdk v1 -> v2 is the lack of documentation, especially with the removal of the DynamoDBMapper. There are some decent docs for changes in other areas of the sdk, but considering the amount of changes in dynamo I was hoping for more there. In fact, I noticed that the removal of DynamoDBMapper was barely noted anywhere under the "difference between 1.x and 2.x" page: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-whats-different.html . Since the enhanced client isn't exactly a full replacement for the mapper, we've had to dive into the mapper code in order to determine exactly what was happening, and how we can replicate that behavior using the enhanced client (specifically, making sure that we correctly replaced DynamoDBMapper.save() calls with the right put/update/create/whatever else using the enhanced client. This is something that I would love to be added to the migration docs: the removal of mapper config SaveBehavior, and what the enhanced client equivalent of each SaveBehavior would be). Adding more to the migration section of the docs that covers this type of thing would be extremely useful, and would reduce the struggle of switching to v2.

To address some of the hurdles you mentioned: Hurdle 1 - I recall seeing a difference between the documentation and actual code behavior for strongly consistent vs eventually consistent reads; however, after testing just now, what you said is accurate. The default behavior for both is eventually consistent reads.

Hurdle 2 - I noticed that "count" as you described it does exist, but I was hoping that there would be a direct equivalent to doing something like DynamoDBMapper.count(class, queryExpression) using the enhanced client. While working through the migration, I had a lot of trouble trying to figure out how to do a Select COUNT in the enhanced client until I realized that there isn't a direct equivalent (unless I'm misunderstanding things, in which case I'd love to have things cleared up). Another point of confusion was that Select COUNT seems to be an option all over the place, including the CLI (https://docs.aws.amazon.com/cli/latest/reference/dynamodb/query.html#options and https://docs.aws.amazon.com/cli/latest/reference/dynamodb/scan.html#options) as well as the Dynamo API (https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#API_Query_RequestSyntax), plus of course being available using the DynamoDBMapper. The fact that it's then not available through the enhanced client really confused me (even if I can do something like add all of the counts from each page, it seems silly to have an actual Select COUNT available elsewhere and not in the enhanced client).

Hurdle 3 - This did also cause confusion, so I'm glad that the issue has been added to the backlog. For some context that might be helpful, the way that I discovered it was by trying to create a table using the enhanced client like this: enhancedClient.table(tableName, TableSchema.fromBean(Item.class)); Where the Item POJO was annotated with various v2 annotations like @DynamoDbPartitionKey, @DynamoDbSortKey, etc. The way the POJO was set up, our setters were formatted like this:

    public T withAttribute(String attribute) {
        this.attribute = attribute;
        return (T) this;
    }

which caused issues with creating the TableSchema, I'm assuming because of the method signature of the setter. Changing the setters to be non-fluent seemed to fix the issue. Maybe it would be nice to have @Getter/@Setter annotations available for situations like this, not sure how feasible something like that would be though.

Thank you again for the response on this issue! Hopefully this feedback is useful as the docs get updated

patrickjamesbarry commented 6 months ago

☝️ what lirons01 said. 100%

patrickjamesbarry commented 6 months ago

Another disconnect to note. When working with a table that has a version attribute/field (@DynamoDBVersionAttribute), v1 DynamoDbMapper would return the new value of the version. In V2, this does not happen. You either have to perform a GET on the new item updated or auto-increment the version yourself to get something that is on par with how v1 behaved.

cenedhryn commented 3 months ago

We now have a DynamoDbMapper section in the migration guide. Hope it's useful, and please give us feedback!

patrickjamesbarry commented 2 months ago

Thank you for providing. Will you be adding Dynamodb encryption migration notes to this page as well, or be linking to a separate page? For example, what is the equivalent to com.amazonaws.services.dynamodbv2.datamodeling.encryption.@DoNotTouch?

debora-ito commented 2 months ago

@patrickjamesbarry the DynamoDB Encryption client is a standalone library, maintained by the Encryption team here -> https://github.com/aws/aws-database-encryption-sdk-dynamodb

There's migration guidance provided in the Database Encryption SDK Dev Guide - https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/ddb-java.html

If you have follow-up questions, please open an issue in the aws/aws-database-encryption-sdk-dynamodb repo.

aws / aws-sdk-java-v2