Add support for new JSON document datatypes

[BCVisin]

https://aws.amazon.com/blogs/aws/dynamodb-update-json-and-more/ http://www.allthingsdistributed.com/2014/10/document-model-dynamodb.html

[BCVisin]

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html

[jlafon]

I'm a little unsure of how to add this to the PynamoDB API. PynamoDB already provides a JSONAttribute for storing JSON - which could use the 'M' data type in DynamoDB. However, the 'M' data type must be encoded into an AttributeValue map, and it ends up looking just like a top level attribute. You end up with an attribute that has nested attributes, making for both an ugly API and an ugly implementation.

Additionally, it would not be backward compatible for people who are already using the JSONAttribute because PynamoDB stores those attributes in DynamoDB as JSON encoded strings.

The same can be said for the BOOL type that DynamoDB now supports. PynamoDB included a BooleanAttribute before BOOL was a data type in DynamoDB. PynamoDB stores boolean values as 1 or 0 (to be compact) encoded into a JSON string, but DynamoDB's BOOL data type uses 'true' and 'false'.

If these datatypes are implemented, it will require a 2.0 release of PynamoDB that is not backward compatible.

[BCVisin]

Ya, I'm not sure either since we just started using PynamoDB.

A major feature we would be looking for is the ability to use Maps and Lists as well as Document Paths

Maybe a new Attribute type names as to keep backwards compatibility?

The nested attributes should probably be handled transparently by the attribute class so that you won't see the AttibuteValue map. It would just be translated based on the python types and return a dictionary.

[dcosson]

Hi, I just started looking at pynamodb and was looking for support for the new document/json fields.

The biggest benefit IMO of using the new native json types as opposed to serializing to a string like the current pynamo json type is the ability to atomically update documents. You can effectively create simple CRDTs for some operations using updates to document paths (e.g. add item to dict without have to read the entire dict first and then re-upload it). If you can structure a table so all inserts & updates work this way, you can use eventually consistent reads for availability & higher throughput without worrying about data integrity which can be a huge win.

For backwards compatibility you could potentially just name it something different than JSONAttribute and leave the string-serialized json attribute, though if you're open to the idea of an incompatible 2.0 release it might keep the API a little cleaner.

[erikcw]

I tend to agree with @dcosson. The map type should be a completely separate Attribute type from JSONAttribute. This would eliminate the backwords compatibility issues as well as preserving JSON persistence for those who may not want to use a map for whatever reason.

[dukedougal]

I'd suggest making it as compatible with the current version of DynamoDB as possible and you should be prepared to break compatibility with old versions of PynamoDB. Compatibility with DynamoDB is more important than compabilityy with PynamoDB.

[joshowen]

I agree that it makes sense to track the latest DynamoDB features in JSONAttribute, and rename the existing class LegacyJSONAtrtribute and list it as a breaking change.

@jlafon @danielhochman any thoughts?

[kapad]

@jlafon Is this in the feature pipeline? Is there any timeline around this feature? Also are you aware of any forks that may have implemented the dynamo document types?

[rixka]

I like pynamodb however due to it's lack of support for Maps and Lists we can't use it.

I sincerely hope that there is an update looking to make use of Maps and List sometime soon.

[kapad]

@RixKa I've made some changes to support Lists and Maps. It's not as good and intuitive as I'd like, but it works to an extent. Would love to get a PR to add full support for Lists and Maps. I don't think there is a plan to add support in this repo.

https://github.com/kapad/PynamoDB

[jmphilli]

hi everyone, @danielhochman and I are working on adding this to a new version of pynamo. we're currently testing an RC so hopefully we can have a release soon.

[jmphilli]

so we added map + list in 2.0.0 -- can we close this?