Define a standard serialization mechanism for arbitrary objects in the language spec

[DartBot]

This issue was originally filed by @Emasoft

---

Dart JSON serialization of objects is not interoperable. An universal default serialization format is urgently needed.

PROBLEM: For a language designed as a new platform for scalable web app engineering, JSON shouldn’t come as an afterthought, hidden somewhere in a library. JSON is everywhere on the web, and Dart should speak to you and listen to you in JSON like it’s its native language. The Dart editor should be able to display a JSON serialization of anything the mouse pointer touches. And to be interoperable this serialization should be standard, at least in the default mode. Two dart web apps should be able to communicate and to exchange any object speaking the same language.

A few weeks ago, I found it surprisingly hard to serialize a simple (plain-old) Dart object like

class Customer {   int Id;   String Name; }

to this (very common) JSON representation

{     "Id": 17,     "Name": "John" } and ended up posting this question on Stackoverflow. It took me a bit by surprise when Google’s Developer Advocate for Dart, Seth Ladd, replied that

"Unfortunately, there’s no universal JSON serialization of objects for all platforms."

Well, that’s true. JSON has been designed as a data interchange format, which doesn’t cover the platform and language specific serialization aspects. BUT DART IS LANGUAGE SPECIFIC. So why there is no standard object serialization rules in Dart, leaving a babylon of different formats to impede the development of the web apps and the interchange of data? There is no reason at all.

PROPOSED FIX: Therefore, I think the simple serialization format above – even if not part of any formal standard – should become the default serialization of a Dart object. So, just typing:

var json = JSON.encode(customer);

should give me the the simple JSON representation of customer by default. I should NOT be required to explicitly implement toJson() or use third-party libraries.

Please give this maximum priority: Dart should learn JSON while it’s still young.

[dgrove]

Removed Type-Defect label. Added Type-Enhancement, Area-Language, Triaged labels.

[DartBot]

This comment was originally written by @Emasoft

---

This is another strenght of Dart that is going to be lost if the developers do not act quickly: Dart has the same classes both in server side code and client side code. THE SAME CLASSES. This means that the same objects can be transferred from the client side to the server side without modifications. Dart should providing a way to serialize in binary format and also in json format all objects in a complete automated way. No annotations or tags should be needed. All Dart objects should be provided by default of a Json serialization option, and a binary serialization method. Also a class to send and to receive objects from client to server should be provided for easy synchronization. Being able to do that would be a huge advantage for Dart over all other languages.

[DanTup]

At my company we're currently evaluating a number of technologies for a future version of our products. I was both surprised (and disappointed) to find that there is no simple way to parse JSON into a Dart class today without a third party library.

Given how common JSON is in web apps, this seems like an insane oversight. Given one of the advantages of Dart is bring types to client side coding, not being able to parse JSON into a type seems like pure madness. Maps are no substitute; they lose all the benefit of having classes, and also don't work the same in DartPolymer (obj['thing'] vs obj.thing).

[gbracha]

Not really a language issue as such. Reclassifying (though maybe its Tools not Libraries).

---

cc @alan-knight. Removed Area-Language label. Added Area-Library label.

[DartBot]

This comment was originally written by Mark.Dec...@gmail.com

---

don't know about this, in java there is serialization too. I totally agree with comment #­3, when we were having a look at dart, the thought of writing your own mapper everytime didn't help to convince those who wanted to use javascript and angular instead of dart anyway..

[DartBot]

This comment was originally written by iko...@gmail.com

---

This feature is strongly needed for real fast work and involving new programmers in Dart! It should be as simple as: var json = JSON.encode(customer);

[alan-knight]

What seems to be being asked for is in general difficult to impossible. That is, if you want

  1) automatically encoding an instance of an arbitrary Dart class as someEncodeMethod(thing)   2) use the "standard" JSON-ish encoding described in the initial bug report   3) be able to de-serialize that data, or data from an arbitrary external source using JSON format into an instance of a class

1) is possible, but extracting the data from an arbitrary instance requires reflection, which is an issue for code size when compiled to JavaScript. We probably do not want to inflate the JavaScript code size to support this for all classes.

2) Even when extracting the data is possible, Dart does not promise to preserve the field names when compiled, so generating a representation that uses those field names require extra information as well as extra code.

3) The format in (2) does not provide enough information to de-serialize into a type. It doesn't specify the type, or how to construct it.

There are variations of this that are possible, some of them are already provided. If you want 1 and 3 but don't care about 2, then the serialization package will work fine either using reflection, a transformer for simple classes, or hand-written rules. It will also handle 1 and 2 without 3. Or 1 and mostly 2 and 3 if you let it provide additional information in the output. But that won't work if it's some external source providing your JSON, without the additional information.

See http://www.dartdocs.org/documentation/serialization/0.10.2/index.html#serialization/serialization_transformer for a description of using the transformer, although the text there doesn't directly describe using the SimpleJSONFormat.

[DartBot]

This comment was originally written by @Emasoft

---

Nothing is impossible. All you need is to define conventions beforehand.

1 ) You don't need reflection, but only a very specific piece of reflective code optimized to extract data from an instance of a class. This is a very narrow function, with very simple requisites. Nothing like full reflection.

2 ) As in C#, you don't need to preserve field names and types for all classes. Only classes explicitly marked with the annotation [Serializable] would automatically preserve all metadata as annotations even when compiled to javascript. It is a very specific functionality, and it is usually applied only one or two classes in a project (those classes that the app need to send back and fourth from client to server in serialized form). You must assume that the same identical Dart class exist in the source code of the server side app and in the source code of the client side app. Any difference should make the deserialization process to fail.

3 ) The JSON format is configurable to store anything we need. And all we need is a way to store a reference to the Dart (or js converted) class already existing in the executing app, both on server and on client. All the type annotations are preserved for Classes compiled with the [Serializable] attribute, so all the JSON file should contain is a unique reference to the serialized class. You can use the name of the class or, to avoid all risks, an SHA3 hash of the class structure. All should be working is the class source code is identical both in the serializer application and in the deserializer application. The same .dart class file can often be shared between the two projects to make sure of it.

As you can see all 3 features can be implemented if some simple requirements are added.

[alan-knight]

We don't have a good way to provide only very narrow reflection. @­MirrorsUsed is difficult to use and getting it wrong has bad consequences. However, we can do it non-reflectively easily enough with a transformer, at least for limited classes.

So if your requirements are that you don't need to consume JSON from external sources and are allowed to add extra information above and beyond the trivial JSON, then this is doable now. Set up the serialization transformer to operate on your_shared.dart file. Import the generated your_shared_serialization_rules.dart.

String write(obj) {    var s = new Serialization()..selfDescribing = false;    rules.values.forEach(s.addRule);    var map = s.write(obj, format: new SimpleJsonFormat(storeRoundTripInfo: true));    return JSON.encode(map); }

read(String s) {    var s = new Serialization()..selfDescribing = false;    rules.values.forEach(s.addRule);    s.read(JSON.decode(s), format: new SimpleJsonFormat()); }

[DartBot]

This comment was originally written by @Emasoft

---

Do you mean that every time I change my classes, I need to rewrite the serialization rules? This doesn't make any sense. A serialization process should be automatic. If I need to write each time the rules, It would be quicker to write directly the json file myself! All objects derived from classes declared as serializable should provide a .serializeToJson() method, returning the json serialization of the object, with no other external "rules" file to mess with. When I want to share with another developer or with an add-on developer a certain object, all I have to do is to give him the class .dart file. If it is not minified, I don't even need to do that, because the dart script is already included in the web app and loaded in his browser, he just need to save it from Chrome inspector. I see no problem with this, all you need is to add a few new instructions to the Dart specifications.

[alan-knight]

It's a transformer, so it happens automatically.

[DartBot]

This comment was originally written by @Emasoft

---

Maybe I wasn't clear enough: I don't want to write a transformer. I don't want to write anything. I need a default serialization method available for all objects, providing me a standard serialized version of any object. This method should be called at runtime and providing to me a json string that can be stored locally, saved or sent to any other dart app and then deserialized just as easily with a single method call. Why I need to write a transformer for a feature that should be already present in the Dart language specifications and available to all?

[alan-knight]

The transformer is already written. What you would need to do is add something to the pubspec once to make sure it is invoked.

I conjecture that the feature is not present in the Dart language specification because they are focused on minimizing code size and it would require retaining additional information that is otherwise not present at runtime.

I've provided an example of a mechanism that requires three lines of pubspec configuration, and with slightly refactoring only requires an import and one line of setup to read/write objects in "standard" format. That seems fairly close to me without requiring re-organization of the mirrors library or imposing overhead on all objects.

[DartBot]

This comment was originally written by @Emasoft

---

Maybe you should publish on GitHub a working example of this, because I don't see how that can be done without changing the Dart specifications, without having to write custom transformations rules for different classes, and without rewriting them anytime I change my code.

[alan-knight]

https://github.com/google/serialization.dart/tree/master/test/transformer

[DanTup]

I haven't tested it yet; but this solution seems reasonable. However:

Disclaimer This is not an official Google project.

This makes it sound somewhat unsupported. Surely if something as fundamental as serialisation isn't going to be included (and I still think the arguments for this are very weak), it should at least be a fully supported package?

[DartBot]

This comment was originally written by @Emasoft

---

Great. I'll make some tests serializing some classes from my projects, and'll get back to you tomorrow.

[alan-knight]

This sort of thing could also be extended in a few interesting directions.

The serialization setup calls could easily be simplified, and probably should be.

Right now the additional information is saved out as "rule" : <ruleNumber>. The transformer could generate the name of the class in a way that was more guaranteed to be unambiguous (e.g. include the library and/or package as part of it) and then we could store it as something like "class" : <className> which is more robust if the list of rules changes and more readable, though less compact.

If the concern is instead with interop and de-serializing things that don't have the additional information there at all, there are things we could do as well. For example, we could have a mixin that provided a toJson and fromJson method. The toJson would almost always be automatic. For fromJson the user might have to supplement the information if a variable has another serializable class in it. But it could look something like

class Stuff extends Object with OurMixin {    var a, b, c, d;   fromJson(json) {      var instance = super.fromJson(json);      instance.myclass = new MyClass().fromJson(json['myclass']);   } }

This deals with the two issues that the format doesn't tell us that those values are supposed to be a MyClass instance, and that in order to create one (without mirrors) we need some explicit code somewhere that calls a constructor for MyClass. But this is only needed for fields that are not directly representable in JSON. It's not perfect, note that we're creating a myClass and throwing it away, because we want to inherit a default implementation, and we can only do that for instance methods. But it does let the user handle arbitrarily complicated cases, invoking non-default constructors, using generics, or de-serializing to non-standard collection types, e.g. ObservableList.

[DartBot]

This comment was originally written by esnellman2@gmail.com

---

A year after 1.0 we still do not have Refactor>>Format in the IDE...
What you are really looking for is a Source>>Generate JSON constructor/toString from fields.

Eclipse Java development tools have these features today if you want to witness how they function.

[DanielJoyce]

Serialization lib produces terrible json. It's not something you would want to expose as a third party api because every other tool out there likely can't product the format.

Yes other languages don't have json as a part of core but they require a lot less setup and are a lot more configurable. Dear needs a json library that produces user friendly readable json right out of the box.

[munificent]

We have a number of serialization schemes suited to different uses now: https://www.dartlang.org/articles/libraries/serialization

It may be that each of those aren't sufficient for various reasons, but I don't think this issue is precise enough to get us there, so I'm going to close it. If you run into serialization cases that aren't handled well, please do file an issue with details.

[devxpy]

I wonder if something like python's pickle can be designed for dart...

[Keithcat1]

Maybe once macros are added to the language? I'm having trouble finding a binary serializer, let alone one that can handle multiple references to the same object without copying it or doesn't require me to manually write the serialization code or run a command every time I change my code (I'll definitely forget and then wonder why it isn't serializing properly). In short, I miss pickle.