Open kiritsuku opened 2 years ago
Thank you for reaching out @kiritsuku. We see how we can improve the interface it more user-friendly. This was added to our backlog, but its priority is not high.
For anyone who'd also like to see us support this feature, please add a 👍 to the original description.
@kiritsuku One of the ways I found is to extract primayKey and sorkKey from lastEvaluatedKey and create custom object with these values, which can be serialized and sent back to web.
Once we get back the serialised page/pointer we can de-serialize it and extract the pk and sk and re-construct the lastEvaluatedKey with these two vaues as PK and SK.
This approach works because at the end of the day Dynamodb only cares about PK and SK so other fields can be ignored.
@chandrasekharpatra You are right, just serializing PK and SK is enough. In fact in some cases it is even enough to just serialize the SK since the PK may already be known to the application. That simplifies the serialization and deserialization of the marker for the next page quite a bit.
@kiritsuku Thanks for the Code Example of Serializing and Deserializing LastEvaluatedKey
I just hit this same issue trying to create pagination token String that I can pass back and forth between and app and client. The plan was to use Gson to convert the Map<String, AttributeValue>
key from DynamoDB into JSON, and then Base64 encode. But the LastEvaluatedKeys
after the encode/decode round trip were rejected by DynamoDB as an exclusiveStartKey
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The provided starting key is invalid: One or more parameter values were invalid: An string set may not be empty
I think the root of the problem is that AttributeValue
is built to check the class of lists/maps such as ns
, bs
, ss
to tell if they hold values. The methods like hasNs()
have logic like this (from decompiler).
public final boolean hasNs() {
return this.ns != null && !(this.ns instanceof SdkAutoConstructList);
}
When Gson constructs an AttributeValue
from JSON it uses ArrayList
instead of SdkAutoConstructList
.
Is it possible to change the logic in hasNs()
and similar methods so they are not dependent on the class of the List? That would make encoding/decoding JSON much easier. As a workaround, I've written a custom AttributeValue
JSON deserializer that checks the type
value and then creates the AttributeValue
using builtin methods like fromS()
.
BTW, just serializing the PK and SK is not a universal solution. In my case I'm using a GSI and the lastEvaluatedKey
from DynamoDB included both the PK and SK of the GSI AND the PK from the table's index.
Since May 2023, AWS's Java 2.x SDK includes an Enhanced Document API that simplifies converting pagination tokens between the AWS SDK's objects and JSON strings that can be passed over HTTP.
AWS blog post demonstrating use cases: https://aws.amazon.com/blogs/devops/introducing-the-enhanced-document-api-for-dynamodb-in-the-aws-sdk-for-java-2-x/
Sample code for converting between Map<String, AttributeValue>
pagination tokens and JSON strings:
import software.amazon.awssdk.enhanced.dynamodb.document.EnhancedDocument;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import java.io.UncheckedIOException;
/**
* Convert a JSON string representation of a DynamoDB pagination token to the format required by DynamoDB API calls.
* @param paginationTokenJson JSON string representing the last paginated API call's last evaluated record key
* @return Map<String, AttributeValue> exclusive start key for the next paginated DynamoDB scan/query API call
* @throws UncheckedIOException exception thrown if fail to parse pagination token
*/
public Map<String, AttributeValue> convertToExclusiveStartKey(final String paginationTokenJson) throws UncheckedIOException {
return EnhancedDocument.fromJson(paginationTokenJson).toMap();
}
/**
* Convert a DynamoDB attribute value map to a JSON string.
* @param attributeValueMap DynamoDB item key represented as a map from attribute names to attribute values
* @return String JSON string representation of the DynamoDB item key
*/
public String convertToJson(final Map<String, AttributeValue> attributeValueMap) {
return EnhancedDocument.fromAttributeValueMap(attributeValueMap).toJson();
}
Expanded discussion at https://www.marksayson.com/blog/serializing-deserializing-dynamodb-pagination-tokens/.
Can likely resolve this issue given the new classes that remove the need for custom serialization/deserialization code for DynamoDB pagination tokens.
Wow that code snippet above is a lifesaver! Thanks
Describe the feature
In order to use paged scanned results in DynamoDB one has to rely on the LastEvaluatedKey field of the scan response. However, in the Java API the raw JSON response can't be accessed, we only can access it through its Java type
Map<String, AttributeValue>
. This makes it quite cumbersome to serialize and deserialize the LastEvaluatedKey to a string that can be forwarded to a client, which then in return can ask for more elements if wanted.Serializing the Java type was already discussed here: https://github.com/aws/aws-sdk-java-v2/issues/2295 It can be done with:
The deserialization of this key is more complex:
The issue is that the type
AttributeValue.BuilderImpl
is not publicly visible. Making it public would make deserialization a lot shorter and also more future proof, since theBuilderImpl
is not part of the API right now and therefore would break this code if changed. Creating theAttributeValue
manually without relying on bean-based deserialization would also be quite a hassle.Use Case
The use case is deserialization of a LastEvaluatedKey formatted as JSON to the Java type
Map<String, AttributeValue>
Proposed Solution
Make
AttributeValue.BuilderImpl
public.Other Information
No response
Acknowledgements
AWS Java SDK version used
2.17.201
JDK version used
11
Operating System and version
Ubuntu 20.04.4 LTS