Open Veetaha opened 3 years ago
This is an awesome deep dive into what's needed in a DynamoDB high level library. Thanks so much for pulling this together!
:wave: Hey, primary maintainer of serde_dynamo here.
We started a new branch that supports aws-sdk-rust: serde_dynamo@3.0.0-alpha.0 and it seems to work fine in the limited testing I've done. As aws-sdk-rust gets closer to being release-ready, I'll put more effort into it, too.
Anyway, I'm just chiming in to say hey, we're still here, we'll help out as we can, etc. etc.
Our business currently uses 5β10 rusoto libraries and lambda_runtime. DynamoDb is our core data storage, which is why we created and maintain serde_dynamo
. I'm pretty sure we'll take a nice, long, favorable look at switching to aws-sdk-rust once it's appropriate. So whether serde_dynamo
gets rolled in or continues to be independent, I'm pretty sure it will support aws-sdk-rust.
Awesome, took a quick look through the implementation. I suspect that https://github.com/awslabs/smithy-rs/discussions/387 might be relevant? I wonder if we should also add the into_xyz
methods you have, (or impl TryInto
?) Anyway, if there are methods the SDK could add to make your life easier, please let us know :-)
Is anything being done on this?
Hey @nuvacore, not currently. We are considering proposals for high-level libraries like this but not until after we GA (which we haven't set a date for.)
Community Note
Terminology
ODM - object-document mapper
Problem
There currently is a low-level DynamoDB client implemented in the SDK, it works with opaque
AttributeValue
types and direct DynamoDB APIs https://github.com/awslabs/aws-sdk-rust/blob/7e43b19fd6fcc753bf5ceff4b2f5d13f6db799d8/sdk/dynamodb/src/model.rs#L6206-L6277This low-level SDK crate provides no convenience APIs (batteries) that simplify the regular idiomatic (Best Practices?, single table design?) usage of DynamoDB.
Working with raw
AttributeValue
s, and ad-hoc implementing common workflows is very inconvenient and error-prone.Solution
I propose we add a new crate that wraps
aws-sdk-dynamodb
low-level library and exposes the following "batteries-included" APIs (the list can be extended):ODM
Implement serialization and deserialization (i.e. object-document mapping) of strongly-typed structs and enums (both plain and discriminated unions) into
aws_sdk_dynamodb::AttributeValue
via proc macros.We can use
serde
to do the bulk of the job. I recommend taking over the job done inserde_dynamo
crate, and also learn the approachesdynomite
crate does.The latter crate is more popular, but it is not very actively maintained. However, from my viewpoint, abusing
serde
as much as possible would be a better approach than implementing proc macros for converting betweenAttributeValue
and strongly-typed structs and enums by hand, but that's debatable.Condition and update expression builder
Condition expressions are used in queries and scans, and update expressions are used in update operations, and they both use custom DynamoDB syntax. It's okay to use raw strings with the standard Rust
format!()
macro for simple cases, but sometimes expressions are very dynamic and the expression might depend on lots of different variables and conditions.Building raw condition expression syntax dynamically is very error-prone, the high-level wrapper crate should expose builders for expressions. See TypeScript's implementation of this concept in
@aws/dynamodb-expressions
package onnpm
.Projection expression utilities
Add some methods, maybe proc macros to generate the types that represent a projection of different combinations of attributes. This requires some more thorough design, but the core problems to solve here:
AttributeValue
The API might look something like:
Pagination utilities
Implement methods for streaming pagination (see
dynomite::DynamoDbExt
to learn about existing implementations).Other utilities for best practices and common workflows
Implement helper utilities according to AWS docs for DynamoDB best practices and single table design.
For example, we've implemented some proc macros for segmented identifiers (described in single table design) in our private repo. We plan to open-source this code, and we may do it earlier to facilitate the development of the high-level DynamoDB crate.
So the list might also include:
Additional context
This issue is nowhere an exhaustive description of the desired design for the high-level wrapper crate for
aws_sdk_dynamodb
, the ideas should be refined and probably extended. However, I think it might be a good starting point to begin the discussion and initiate the work on the MVP subset for the planned high-level APIs (e.g. start with only ODM feature and iterate from that next). We may decide to separate the described crate to other repo and split the planned features described here into more fine-grained issues if this makes sense to the maintainers.Waiting for your feedback!