Example of a simple JSON system using reflectable

[sethladd]

Users have been asking for an easy way to do simple JSON decoding for years. Hopefully reflectable can deliver this.

Here is the use case:

class Person {
  int age;
  String name;
  List<Address> addresses;
}

class Address {
  String street;
  String zip;
}

// Assumptions: no key is used for each top-level object (map), the developer knows what the top-level
// maps are. The structure of the maps is expected to match the structure of the class.
final data = '''
[
  {"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443"}, {"street":"555 second", "zip":"99999"}]},
  {"age":3, "name":"Alice", "addresses":[{"street":"222 cedar", "zip":"23456"}]}
]
''';

We should really make it simple to get a List<Person> from the data string, without code bloat when compiled to JavaScript, and without a source-gen step.

[alan-knight]

Nitpicking, but how do you expect this example to know to create a new Address from the data under 'addresses'? You can leave the types off of every other field and it would be fine, but List can't work here, at least not with "simple JSON".

[sethladd]

Hi Alan,

In this particular case, I would expect my deserializer to see that Person has a List of addresses. The deserializer should create a new list, and then create new Addresses using the default constructor of Address.

I'm comfortable adding an annotation (as is our customer that I'm chatting with on G+). For example, this would be fine:

@serializable
class Person {
  int age;
  String name;
  List<Address> addresses;
}

[alan-knight]

My point is, how does it know to create an Address? You have not told it anywhere that the list "addresses" can be expected to contain addresses.

[sethladd]

Through magic! :)

Ah... I see. OK, we'll have to do List<Address> addresses. Good catch, thank you!

[jmesserly]

Hey Seth, how would you expect to call the serializer/deserializer? Maybe something like this?

// this is a sketch of hypothetical apis using serialization pkg and reflectable
import "package:serialization/json.dart"
    show serializable, JsonDeserializer;

@serializable
class Person {
  int age;
  String name;
  List<Address> addresses;
}

@serializable
class Address {
  String street;
  String zip;
}

main() {
  // The generic lets us have the return type for "from", and also lets us
  // pass the expected type List<Person>.
  var people = new JsonDeserializer<List<Person>>().from('''
[
  {"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443"}, {"street":"555 second", "zip":"99999"}]},
  {"age":3, "name":"Alice", "addresses":[{"street":"222 cedar", "zip":"23456"}]}
]
''');
}

For illustration purposes, something like this would be in the serialization package:

class JsonDeserializer<T> {
  T from(String json) {
    var obj = JSON.decode(json);
    ClassMirror mirror = serializable.reflectType(T);
    // <get default constructor, make instance, set fields ... recursively>
    return result;   
  }
}

class Serializable extends Reflectable {
  const Serializable() : super(/* ..constrain to capabilities we use ... */);
}

const serializable = const Serializable();

[jmesserly]

Sorry that was a wall of code. The main thing I wanted to call out was new JsonDeserializer<List<Person>>() as a way of providing the expected type. As Alan notes, this is very crucial to do in some way.

[jmesserly]

(btw, if you don't mind, I added "dart" after the triple backticks in your code examples, to get pretty colors :rainbow: )

[sethladd]

We'd be happy to have to do:

JsonDeserializer<List<Person>>()

or even something like:

JsonDeserializer(create: () => new List<Person>())

I'm flexible here.

w.r.t. ClassMirror mirror = serializable.reflectType(T); ... I assume this is via reflectable and thus will not require MirrorsUsed or bloat the code?

[sethladd]

(Perhaps this is the wrong forum for a long discussion on serialization. but hopefully reflectable can enable simple JSON serialization. It's a good use case.)

[jmesserly]

or even something like: JsonDeserializer(create: () => new List<Person>())

the issue with that is you'd need ctors for each thing, so it would be more like: JsonDeserializer({ Person: () => new Person(), Address: () => new Address() })

(that's a Map<Type, ConstructorFunction>) ... starts to feel boilerplate-y.

w.r.t. ClassMirror mirror = serializable.reflectType(T); ... can we do this without code bloat or MirrorsUsed?

Should be able to. The idea behind reflectable is that you get mirrors by specifying their capabilities, so the mirror only brings in what it needs. What this does mean is, in my sketch above, every @serializable type would potentially be retained: its default constructor and all of its fields.

It would be awesome if the implementation could follow the reflectType(T) and based on the generic instantiation, conclude that only List, Person, and Address are constructed -- not other serializable types. Not sure how tricky that would be. But still, even if all serializable types are retained (default ctor+fields), that would still be a big help IMO.

(Perhaps this is the wrong forum for a long discussion on serialization. but hopefully reflectable can enable simple JSON serialization. It's a good use case.)

Agreed!

[sethladd]

For:

JsonDeserializer({ Person: () => new Person(), Address: () => new Address() })

Yeah, that's not great. The system should auto-detect the default constructor for a type.

[sethladd]

For research, here's some references about serialization in Dart:

• http://stackoverflow.com/questions/27135876/state-of-serialization-and-deserialization-of-json-in-dart
• https://www.dartlang.org/articles/serialization/

[eernstg]

I'm sure these things could be done using Reflectable, because that should be very similar to an approach based on dart:mirrors. However, it may be more straightforward to use a dedicated transformer, because the kind of code that we could generate in a Serializable transformer (or maybe that's already done?) would be so simple:

import 'dart:collection';
import 'dart:convert';

@serializable
class Person {
  int age;
  String name;
  List<Address> addresses;
  String toString() => "$name, age $age, living at $addresses";
}

@serializable
class Address {
  String street;
  String zip;
  String toString() => "$street, $zip";
}

// Assumptions: no key is used for each top-level object (map), the
// developer knows what the top-level maps are. The structure of the
// maps is expected to match the structure of the class.
final data = '''
[
  {"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443"}, {"street":"555 second", "zip":"99999"}]},
  {"age":3, "name":"Alice", "addresses":[{"street":"222 cedar", "zip":"23456"}]}
]
''';

// Generated by the Serializable transformer
Address mapToAddress(HashMap map) {
  Address address = new Address();
  address.street = map['street'];
  address.zip = map['zip'];
  return address;
}

// Generated by the Serializable transformer
Person mapToPerson(HashMap map) {
  Person person = new Person();
  person.age = map['age'];
  person.name = map['name'];
  person.addresses = map['addresses'].map(mapToAddress).toList();
  return person;
}

void main() {
  JsonDecoder decoder = new JsonDecoder();
  List objects = decoder.convert(data);
  Iterable<Person> persons = objects.map(mapToPerson);
  print(persons);
}

The question is whether there would be a need for more flexible approaches (deserializing into instances of classes that are not statically known to be deserialization targets), but that wouldn't be easily covered by Reflectable, because Reflectable reflection requires metadata (@myReflectable) on the target class.

I'll take a look at the references that came up 3 minutes ago now. ;)

[sethladd]

A transformer is unfortunately not a sufficient solution.

• Transformers require a use of pub build or pub serve.
• Transformers force generated code to be stored in pub serve's memory, which is a deal-breaker if anyone wants to debug their app.
• The API for transformers is not ideal for code generation.

Our users want a simple JSON solution that doesn't require: excessive annotations (a single annotation is sufficient), explicit and manual code generation, code bloat. It should "just work".

I'm very hopefully the reflectable package can provide a solution here. :)

[eernstg]

Reflectable uses a transformer, so if that is a show-stopper then Reflectable cannot be used. With a transformer it should be possible to read/run/debug the transformed code by doing pub build --mode=debug some-directory and then looking into build/some-directory/.. where the transformed files are stored. (But nobody would ever want to debug their app, of course ;-). True, the API for transformers is not convenient for code generation, but it can be done and it would not affect the users.

Still, Reflectable might be an alternative to dart:mirrors in existing approaches to serializability. If there are some good solutions that just fail to be applicable because they rely on dart:mirrors, it might be helpful to use Reflectable rather than dart:mirrors.

[sethladd]

@eernstg I know this isn't your problem to solve, but the pub build --mode=debug some-directory workflow isn't sufficient. It doesn't work well with our IDEs and editors, because they are looking at the source directory. It puts all the generated code into a completely separate directory, and creates a parallel source world.

Is the transformer required even with Dartium?

cc @kevmoo who has a source_gen project that might be a better fit.

[alan-knight]

I thought the point was that Reflectable is a transformer right now, when it can be experimented with, but would be ultimately incorporated into dart2js code generation. In Dartium it can just fall back to using mirrors.

[jmesserly]

I thought the point was that Reflectable is a transformer right now, when it can be experimented with, but would be ultimately incorporated into dart2js code generation. In Dartium it can just fall back to using mirrors.

yeah, I thought so too

[eernstg]

Of course, you never need to consider the output of a translation process if the source language has a well-understood semantics and the translation has a high quality (and tools can consistently support debugging etc. in terms of the pre-translation program). But it is a non-trivial requirement on the tools that they must make Reflectable translation transparent (such that the programmer can use tools showing the pre-translation source program and never even consider the post-translation code). We won't get immediately..

[eernstg]

If it is acceptable in a given context to consider mirrors produced by dart:mirrors APIs as black boxes, then the static mirrors generated by Reflectable might be treated in a similar way, and in that case the programmer would get to see the interfaces (package:reflectable/mirrors.dart, near-identical to the interface of dart:mirrors), and the actual implementation classes would be equally black-boxy with dart:mirrors and with Reflectable. Is that what you are thinking about?

[alan-knight]

Yes. My mental model of Reflectable is that it provides a way to specify what aspects of reflection you want, but that then your usage is the same as if you were using mirrors. That includes being black-box in that they don't require the use of pub serve or build.

On Tue, Mar 10, 2015 at 10:27 AM Erik Ernst notifications@github.com wrote:

If it is acceptable in a given context to consider mirrors produced by dart:mirrors APIs as black boxes, then the static mirrors generated by Reflectable might be treated in a similar way, and in that case the programmer would get to see the interfaces (package:reflectable/mirrors.dart, near-identical to the interface of dart:mirrors), and the actual implementation classes would be equally black-boxy with dart:mirrors and with Reflectable. Is that what you are thinking about?

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/reflectable/issues/4#issuecomment-78104000.

[eernstg]

About "[the Reflectable transformer] puts all the generated code into a completely separate directory": If we want to have transformer-specific tool support, then it would probably not be impossible to offer the choice in IDEs etc.: "Debug the transformed program" versus the normal "Debug the program". With Reflectable, the pre-transform program should have exactly the same semantics (apart from the different implementation of operations in dart:mirrors based pre-transform reflection and static mirror based post-transform reflection), so with a well maintained Reflectable library it should be a matter of working in the pre-transform world and then checking that it works the same in the transformed world.

[eernstg]

About "usage is the same": Nearly. It differs in little things like 'myReflectable.reflect(o)' rather than 'reflect(o)'. But it should be easy to port.

[sethladd]

the pre-transform program should have exactly the same semantics

This is a great goal, but there will always be weird cases where semantics don't match, especially when compiling to JavaScript. Especially when building for the web and all the various different browser nuances, being able to inspect the generated code and step through it is critical.

[alan-knight]

Seth: Does that mean you're saying you require a solution that generates Dart code ahead of time? And, presumably, that you find mirrors unacceptable in principle, even if they worked fine with dart2js tree shaking? It seems like Reflectable, in any implementation, wouldn't fit with your requirements if that's true.

On Tue, Mar 10, 2015 at 10:57 AM Seth Ladd notifications@github.com wrote:

the pre-transform program should have exactly the same semantics

This is a great goal, but there will always be weird cases where semantics don't match, especially when compiling to JavaScript. Especially when building for the web and all the various different browser nuances, being able to inspect the generated code and step through it is critical.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/reflectable/issues/4#issuecomment-78110300.

[sethladd]

If it works with Mirrors for Dartium/VM, that's fine, as long as any code bloat is removed before we compile to JavaScript.

We'd also have to ensure that source maps work excellently well, when a developer is debugging their app in various browsers. The source map should point the user back to code they can see and inspect in their editors.

[jimsimon]

With regards to constructors, it'd be nice if no constructor is called at all. The idea behind serialization (in most cases) is to store state of an object and restore the exact same state at a later time. Since constructors can have logic, executing them could alter the state of the restored object. I know that GSON (Java) uses Unsafe.allocateInstance to accomplish this. I'm curious if the same thing could be accomplished in Dart. I assume the problem would be with accomplishing this in JavaScript via dart2js.

For what it's worth, a first party mirrors based approach for Dartium and the VM sounds awesome. I'd also prefer to see the conversion to JavaScript happen in dart2js instead of a transformer. Having to rely on pub serve can be annoying if your app is set up to serve its own content and you want to do development for it when it's running that way. This scenario forces you to have to stop, build, and run again manually. Note that pub serve is wonderful when doing client only development.

[alan-knight]

Regardless of whether it's a good idea or not, Dart does not have a way of allocating an object without calling a constructor.

On Tue, Mar 10, 2015 at 4:01 PM Jim Simon notifications@github.com wrote:

With regards to constructors, it'd be nice if no constructor is called at all. The idea behind serialization (in most cases) is to store state of an object and restore the exact same state at a later time. Since constructors can have logic, executing them could alter the state of the restored object. I know that GSON (Java) uses Unsafe.allocateInstance to accomplish this. I'm curious if the same thing could be accomplished in Dart. I assume the problem would be with accomplishing this in JavaScript via dart2js.

For what it's worth, a first party mirrors based approach for Dartium and the VM sounds awesome. I'd also prefer to see the conversion to JavaScript happen in dart2js instead of a transformer. Having to rely on pub serve can be annoying if your app is set up to serve its own content and you want to do development for it when it's running that way. This scenario forces you to have to stop, build, and run again manually. Note that pub serve is wonderful when doing client only development.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/reflectable/issues/4#issuecomment-78167220.

[eernstg]

Any approach that involves generated code could generate a constructor for each @serializable class that takes a representation of an object state and creates an object in that state.

[jimsimon]

I think my last comment got a little carried away. The important part of what I was trying to say was that it'd be ideal to avoid side effects when restoring state.

Generating a constructor is fine for output from dart2js, but I'd personally prefer the default constructor and mirrors approach when it comes to Dartium and the VM. I think the default constructor approach is common enough in other libraries (at least in java) that as long as it's documented it won't be a problem.

Also I'm assuming this won't be able to handle properties with type "dynamic" since the type at runtime could be anything, right?

[eernstg]

@jimsimon What makes you think that default constructors will play nice? ;-) For instance, objects with final fields will require them to be initialized during construction, and the default constructor may not choose the values you need (and dart:mirrors will say "..has no instance setter.." if you try to correct them post-hoc).

[alan-knight]

In Dart not only can the default constructor have side effects, so can setting fields. And doing it through mirrors does not give you away around that, unless you go through some gymnastics to get at the private underlying fields.

But this is talking about extremely restricted objects, and the restriction that they can be effectively recreated by calling a default constructor and then setters on fields whose names correspond to the map entries is a reasonable assumption, and less than some. And yes, they can't handle entries typed dynamic, or whose generics are dynamic, or whose types are not completely concrete (i.e. where the values might be things that implement the interface or are subclasses), the object graph is acyclic, and probably other restrictions as well.

On Wed, Mar 11, 2015 at 6:17 AM Erik Ernst notifications@github.com wrote:

@jimsimon https://github.com/jimsimon What makes you think that default constructors will play nice? ;-) For instance, objects with final fields will require them to be initialized during construction, and the default constructor may not choose the values you need (and dart:mirrors will say "..has no instance setter.." if you try to correct them post-hoc).

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/reflectable/issues/4#issuecomment-78260542.

[jimsimon]

Is this package far enough along that I could try spiking out a serializer/deserializer this weekend?

Also, it may be possible to handle dynamic types as lists and maps based on whether the property's value starts with a { or a [. It's not a perfect restoration of the original object, but it could be a workable fallback. Either falling back to lists and maps or erroring out and saying "dynamic types aren't supported" work for me.

Other serializers have ways of handling polymorphism, but a 1.0 release of this serialization library wouldn't necessarily have to. Some of these methods might allow dynamic types to work as well. See http://programmerbruce.blogspot.com/2011/05/deserialize-json-with-jackson-into.html for one way to handle this.

[eernstg]

No, unfortunately you cannot test anything like a serializer/deserializer with the current Reflectable package, it is not sufficiently complete for that. But we are working on it all the time. ;)

[re222dv]

Polymophism would be necessary for me, it's easy to do with dart:mirrors by storing the library and class in the JSON, and then using currentMirrorSystem().findLibrary(librarySymbol).declarations[typeSymbol]. Would this use case be possible with reflectable?

[eernstg]

Trying to recover the context: I'm not sure what it means to 'need polymorphism' in the context of (de)serialization, but I suppose it might mean "I want to obtain support for (de)serializing objects which are instances of subtypes of a given type T without giving annotations or similar explicit directions concerning any other types than T" (so (de)serialization for the subtypes should be handled automatically). I would never expect the receiver of a serialized value to know statically which type of object it is receiving on a given socket or similar input channel (that should be specified in the serialization itself), but Dart's compilation model makes it reasonable to require that it must be an instance of a class C which is a member of a statically known set of classes. That set could be enumerated directly (by having annotations on every member of the set), or it could be obtained by quantification (as in "please support serialization for all subtypes of C"). So I'll assume that we are talking about the situation where librarySymbol and typeSymbol belong to statically known finite sets of symbols; typically we'll just support all libraries in the program, but only some of the types.

With that, you would have to use librarySymbol and typeSymbol values belonging to the set that you have support for; other than that, there is nothing stopping you from doing

  myReflectable.findLibrary(librarySymbol).declarations[typeSymbol]

to get a ClassMirror for the class named typeSymbol in the library named librarySymbol. We haven't yet implemented anything in the area of class mirrors, but I don't foresee any particular difficulties in supporting them just like dart:mirrors, that's actually just a lot of static information delivered using statically generated maps etc.

[Pajn]

Yes I understand that the libraries and classes available must be annotated. The use case I'm thinking about is if there are multiple subclasses of Address and addresses contains a mix of those.

@serializable
class Address {
  String street;
  String zip;
  String country;
  String toString() => "$street, $zip, $county";
}

@serializable
class StateAddress extends Address {
  String state;
  String toString() => "$street, $zip, $state $county";
}

final data = '''
[
  {"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443", "country": "SE", "@type": "Address"}, {"street":"555 second", "zip":"99999", "country": "US", "state": "OH", "@type": "StateAddress"}]}
]
''';

After I have deserialized that JSON I would expect a list with one instance of Address and one instance of StateAddress so that toString() prints the correct format.

[eernstg]

That's exactly what I meant: 'that [i.e., the type of the incoming object] should be specified in the serialization itself' is satisfied in your example by the bindings on the form "@type" : "SomeClass", so you would certainly get an instance of Address and an instance of StateAddress from the deserializer, as long as the serialization package you are using will provide and interpret these bindings.

There is no "@type": "Person" binding in the top-level element in data; if you want to discover that this is a Person without adding such a binding we're in a different setting, but I assume that it was omitted by accident.

[Pajn]

If you want to discover that this is a Person without adding such a binding we're in a different setting, but I assume that it was omitted by accident.

The point was the Address classes so I ignored it, might have been dumb tough as it causes more confusion.

[eernstg]

OK! Generating and interpreting those "@type": "SomeClass" bindings is handled by some serialization library L in any case, so it's only relevant to the design of Reflectable to the extent that L needs something very special from Reflectable in order to do that (which does not seem likely).

[sigurdm]

I have added a small serialization example, showing what we hope to achieve (https://github.com/dart-lang/reflectable/blob/master/test_reflectable/test/serialize_test.dart, https://github.com/dart-lang/reflectable/blob/master/test_reflectable/lib/serialize.dart). It already runs in the untransformed case (relying on dart:mirrors) and I hope to have added enough functionality to the transformer very soon (https://codereview.chromium.org/1181153003, https://codereview.chromium.org/1182083002, https://codereview.chromium.org/1181413005) that it can also run transformed. However I am running into trouble with the way transformers only work on one package at a time (https://github.com/dart-lang/sdk/issues/17306). So it is currently not possible to write have the Serializable extends Reflectable defined in one package, and then use it in another. Hopefully this will improve when https://github.com/dart-lang/sdk/issues/17306 is resolved.

[eernstg]

Thanks for all your input! We'll close this issue now. In case the test serialize_test.dart does not adequately embody the requirements coming from this application area it should be raised as a new issue.

[jmesserly]

LGTM!

(For those interested, here's the link for test_reflectable/lib/serialize.dart and test_reflectable/test/serialize_test.dart)