Enhancement: @Json or @Serialize(format=json) Annotation

[stickfigure]

Original issue 230 created by objectify on 2015-03-20T15:14:19.000Z:

I am missing an annotation, which allows me to serialize and store an object graph using Json instead of Java Serialization.

Json is much more flexible, tolerant and faster, and I would love to see it in objectify natively.

Details:

1. Java Serialization is opaque, Json is human readable.
2. Java Serialization typically takes longer and produces a larger file than Json.
3. And most important: The main problem with Java Serialization is versioning. By default, it stores the full package name of a class. Even simple refactoring will break it and render the stored data unreadable. Java Serialization may be an acceptable technique for wrapping data short-term (such as for example data that is transferred over REST and immediately unwrapped on the other side), but I don't consider it suitable for long-term data storage (such as data store), as schemas evolve over time.

Don't overlook:

The Json standard does not require maintaining item order in Json arrays. As Java List is an ordered collection, this will cause problems. If we implement Json, we should choose a library which does maintain item order, such as Gson.

[stickfigure]

Comment #1 originally posted by objectify on 2015-03-25T18:59:21.000Z:

With the new embedded behavior in Objectify 5 (thanks to the low-level EmbeddedEntity), there's a lot less need for @Serialize, and generally when you will need it (say, cyclic object graphs) json won't help you either.

I'll leave this in as a feature request though.

I'm not sure what you mean about item order in JSON arrays. Arrays are ordered and Jackson preserves the order. Perhaps you are thinking about Javascript (not JSON) and key order in an object? That doesn't affect us (it actually doesn't even affect Javascript, because all browser impls preserve order).

[stickfigure]

Comment #2 originally posted by objectify on 2015-03-26T02:04:30.000Z:

There is a use case where I use @Serialize List to store an embedded entity within another entity, but don't want objectify to persist the list as embedded in order to save cost/database GETs. I noticed CPU time is lowest when I serialize the complete list. I am doing this frequently when the embedded entity exists elsewhere in the datastore, but I frequently need to access a list of only a few properties of the entities in the list, let's say I mirror the entity as cache. Example: I have a User entity and need a list of users who liked a Picture. I would then mirror the user within the Picture entity with only the properties that I need. Problem with @Serialize is the contract breaks when I add or remove fields. Json could handle that better.

You are right, json.org says "An array is an ordered collection of values." So we don't need to worry about item order. I remember that older implementations of Json array messed up the order, or at least I thought to remember. Sorry about that.

[stickfigure]

Comment #3 originally posted by objectify on 2015-03-26T02:19:32.000Z:

I'm a little confused by your comment. Deserializing an EmbeddedEntity does not produce additional database GETs or read operations. There might be different CPU costs parsing the data via serialization vs json vs protobuf vs objectify, but I would be surprised if the difference dominated your expenses. And I'm not even sure what would win (I haven't benchmarked them).

Regarding adding/removing fields, if you specify a serialVersionUID, java serialization has the same semantic behavior WRT adding/removing fields as Jackson in "forgiving mode" (incidentally, not the default). The major problem with serialization is changing class types. Then again, JSON deserializers have this problem too with polymorphic structures. There's no free lunch.

Again, I'll keep this as a feature request because it's not a bad idea - it just doesn't sound critical. It's fairly easy to implement on your own as a Translator (see SerializeTranslator.java as a guide) so if you want to try it yourself, I'll take a PR. This doesn't require any modification to Objectify.

[stickfigure]

Comment #4 originally posted by objectify on 2015-03-26T02:36:49.000Z:

I didn't want to get too explicit, so I omitted things from my tests, sorry. Also, this was about a year ago and I didn't keep results, but what I remember was this:

• Storing a list of keys and doing batch GETs to retrieve the original entity was most expensive.
• Storing a list of embedded entities (no cost for gets, but kind of CPU intensive due to processing, and also has the disadvantage that memcache may fill up the "missing" properties if cached, I don't remember if on the client or before it is returned).
• @Serialize the list (cheapest, but with all the Java serialization drawbacks).

[I don't have much time right now, but I will look into your suggestion in a week or two hopefully. I need to find a way to store those lists cheap.]

[stickfigure]

Comment #5 originally posted by objectify on 2015-04-02T02:57:23.000Z:

I have worked around this using @OnLoad/@OnSave lifecycle events, where I convert from and to JSON. I would still like to see this feature added.

[petrovskyy]

Specifying serialisation format would definitely be perfect enhancement Any plans to add it in near future?

[maeyan-zero]

With translators this is very easy. The recipe is one new annotation - say "@SerializeJson" and a TranslatorFactory implementation. I think it makes sense to have a different annotation per serialisation routine than to add a format delegator to the current Serialize annotation.

[dincek]

Maybe you could add @Serialize(serializer = MyCustomSerializer.class) and then you would give us all options, this is then our decision what to implement / use. For example only restriction would be that MyCustomSerializer needs to accept and return com.google.appengine.api.datastore.Text value.

[kirillgroshkov]

+1, nice to have feature. Currently using my own JsonSerializable interface and Objectify TranslationFactory for that

[kirillgroshkov]

After few hours I managed to make it work using custom annotation and custom TranslatorFactory. Example is here: https://github.com/kirillgroshkov/objectify-json-serialization