JSON over HTTP

[itsjoeconway]

JSON over HTTP Proposal

Abstract

The purpose of this proposal is to define an API for a productive JSON deserialization in Dart client-side code. The proposed API is not specific to JSON; it could be extended to any form of dynamic data binding. JSON makes a good example. This solution is currently not implementable in Dart 1.x. This solution should include support for all compilation targets.

Consider the result of some HTTP request as a list of JSON objects:

[
  {
    "id": 1,
    "name": "Thing"
  },
  {
    "id": 2,
    "name": "Thingamabob"
  }
]

A productive way of making this request and getting back this list of JSON objects as Dart objects might look like the following:

var request = new Request<Thing>("/things");
var thingResponse = await request.get();
if (thingResponse.statusCode == 200) {
  // type annotation added for clarity, is unnecessary
  List<Thing> things = thingResponse.body.asList();
}

Where a Response.body.asList() is of type List<T> as in Request<T>. T must extend some type that has behavior for deserialization behavior. De-serialization of a single object follows the same pattern, but accessed with asObject().

var request = new Request<Thing>("/things/1");
Thing thing = (await request.get()).body.asObject();

Declaring the parameterized type for Request requires minimal effort. In a simple case, it declares managed properties and extends Codable:

class Thing extends Codable {
  managed int id;
  managed String name;

  String notManaged;
}

Other Codable subclasses can be managed:

class Foo extends Codable {
  managed int id;
  managed Thing thing;
}

Such that the following JSON is deserializable:

{
  "id": "foo1",
  "thing": {
    "id": 1,
    "name": "Thing"
  }
}

As an alternative syntax, all managed properties can be declared as so:

class Foo extends Codable {
  managed {
    int id;
    Thing thing;
  }
}

The name of the key in the payload can be specified with metadata:

class Foo extends Codable {
  managed {    
    int id;

    @CodableKey("thingamajig")
    Thing thing;
  }
}

Motivation

Moving dynamic data into and out of an application is a necessary component of a Dart application. Constraints on meta-programming on most Dart compilation targets require manual or code generated solutions. Manual solutions are tedious and prone to error, while code generation requires deeper knowledge of the platform, extra tooling and therefore deters new users.

A built-in solution to JSON deserialization is an expected feature of any modern application programming language; so much so that Swift modified their compiler to support it. The value-add of a unified mechanism to productively model JSON data as Dart objects is significant. A new user can immediately see results with an existing API (or one they are able to build quickly), which will impact their long-term usage of Dart.

Proposal

The managed keyword is applied to properties and accessors of a type. These properties and accessors are available to the interface of the type, but storage is deferred to some other mechanism. The following is syntactically valid:

class T {
  managed int i;
}

var t = new T()..i = 1;

The base class Object adds a new property, managedMembers, which contains runtime-available metadata for all managed declarations. For AOT targets, this information is created during compilation. This effectively moves code generation into a compiler pre-processing step.

The invocation t.i= is not a simple property assignment, but is resolved to an invocation of:

t.setValueForKey(#i, 1);

Likewise, the getter t.i is an invocation of:

t.valueForKey(#i);

Because T does not currently implement setValueForKey or valueForKey, this implementation must be provided by T or a base class. Codable implements these two methods as little more than map access:

class Codable {
  Map<Symbol, dynamic> _data = {};

  void setValueForKey(Symbol key, dynamic value) {    
    _data[key] = value;
  }

  dynamic valueForKey(Symbol key) => _data[key];
}

Thus, access to managed properties is always dynamic. (This part is debatable, but it allows for a tiered approach where this functionality is available earlier and can be optimized later without an API change.)

A Codable is instantiated with a Coder. This would require that constructors be inherited; that Dart constructors are not inherited is surprising and requires workarounds for some design patterns. Codable would define two constructors:

class Codable {
  Codable();
  Codable.fromCoder(Coder coder) {
    managedMembers.forEach((k, metadata) {      
      var value = coder.decode(k, metadata);

      setValueForKey(k, value);
    });
  }
}

A Coder is an abstract key-value container object. A simple implementation would wrap a Map:

class KeyedCoder implements Coder {
  KeyedCoder(this._keyValues);

  Map<String, dynamic> _keyValues = {};

  dynamic decode(String key, ManagedMemberInfo metadata) {
    var value = _keyValues[key];
    if (!metadata.isAssignableWith(value)) {
      throw 'invalid value';
    }

    if (metadata.transformation != null) {
      return metadata.transformation(value);
    }

    return value;
  }

  ....
}

(Important: this does not limit concrete Coders from implementing key-lookup in other ways. Additionally, a Coder should not copy data.)

The Request<T> object from earlier deserializes data as such:

class Request<T extends Codable> {
  Request(this.path);

  Future<Response<T>> get() async {
    var response = await http.get(...);

    // Decodes according to content-type
    var decodedBody = await decode(response);
    if (decodedBody is List) {

      return new Response<T>()
        ..statusCode = response.statusCode
        .._objects = decodedBody
          .map((o) => new T.fromCoder(new KeyedCoder(o)))
          .toList();

    } else if (decodedBody is Map) {

      return new Response<T>()
        ..statusCode = response.statusCode
        .._object = new T.fromCoder(new KeyedCoder(decodedBody));

    }
  }
}

This requires that parameterized types can be instantiated and that their constructors are known.

Summary of Proposed Changes

• Adds managed language keyword.
  • Allows for dynamic storage/implementation.
• Adds Object.managedMembers.
  • A limited set of metadata to be generated at compile-time for AOT targets from managed properties.
• Allows constructors to be inherited.
• Allows parameterized types to be instantiated.
• Adds Codable and CodableKey.

[kevmoo]

@joeconwaystk see https://github.com/matanlurey/dart_serialize_proposal

[kevmoo]

@joeconwaystk also see my comment here: https://github.com/matanlurey/dart_serialize_proposal/issues/8#issuecomment-329644856

[natebosch]

I think this issue is covering 2 things:

• Automatic deserializing of data over HTTP calls instead of explicitly grabbing the String or Stream of bytes and deserializing. I think this is probably unlikely to gain much traction.
• Better ability to deserialize JSON to Dart types instead of Map<String, dynamic>. This is something that has been discussed in a number of places and I think we still hope to be able to do something better around this, but it's not clear what that will look like. See for instance #31876