Serialization of constant fields, and extensible discriminators

[HertzDevil]

JSON::Serializable.use_json_discriminator has a few problems:

• The discriminators are associated with some superclass in a hierarchy, and defined within that base type, even though they are a property of the subclasses. (https://github.com/crystal-lang/crystal/pull/8406#issuecomment-548187513)

• The discriminator is considered only during deserialization but not serialization; correctly doing the latter requires defining a read-only instance variable for the discriminator, even though the dynamic type of an object almost always behaves the same as a discriminator. (That exception is when multiple discriminator values map to the same type. I don't think anyone ever does that.) Consider:

  abstract class Shape
  include JSON::Serializable
  use_json_discriminator "type", {point: Point, circle: Circle}
  property type : String
  end
  
  class Point < Shape
  property x : Int32
  property y : Int32
  end
  
  class Circle < Shape
  property x : Int32
  property y : Int32
  property radius : Int32
  end
  
  shape = Shape.from_json %({"type":"point","x":1,"y":2})
  shape                # => #<Point:0x7fdcd005fe40 @type="point", @x=1, @y=2>
  shape.type = "circl" # replacing the discriminator *and* making a typo
  shape.to_json        # => {"type":"circl","x":1,"y":2}

  The discriminator API should disallow this kind of code. Omitting @type altogether also produces an incomplete JSON:

  abstract class Shape
  # deserialization still works without declaring `@type`
  # property type : String
  end
  
  shape = Shape.from_json %({"type":"point","x":1,"y":2})
  shape         # => #<Point:0x7fcb5af81e40 @x=1, @y=2>
  shape.to_json # => {"x":1,"y":2}

  So one must write @type : String or getter type : String, despite this instance variable having no uses outside (de)serialization.

• use_json_discriminator is designed for class hierarchies rather than module hierarchies. This won't work due to an infinite recursion in Shape.new(JSON::PullParser):

  module Shape
  include JSON::Serializable
  use_json_discriminator "type", {point: Point, circle: Circle}
  property type : String
  
  def self.from_json(string_or_io)
    parser = JSON::PullParser.new(string_or_io)
    new parser
  end
  end
  
  class Point
  include Shape
  property x : Int32
  property y : Int32
  end
  
  class Circle
  include Shape
  property x : Int32
  property y : Int32
  property radius : Int32
  end
  
  Shape.from_json %({"type":"point","x":1,"y":2})

  Likewise, disjoint unions cannot use discriminators, because each variant type must then use use_json_discriminator "...", {...: self} and doing so produces a similar infinite recursion.

To address these problems, we first consider the case when there is exactly one concrete type; the discriminator then reduces to a single constant field. This field shall be deserialized and serialized even when a corresponding instance variable does not exist. This is still useful in situations where a JSON format might define a magic constant and require all documents to include that constant. A sensible place to declare the constant field is via the JSON::Serializable::Options annotation:

@[JSON::Serializable::Options(constants: {type: "point"})]
class Point
  include JSON::Serializable
  property x : Int32
  property y : Int32
end

x = Point.from_json %({"type":"point","x":1,"y":2})
x         # => #<Point:0x7f8d332b6e90 @x=1, @y=2>
x.to_json # => {"type":"point","x":1,"y":2}

Point.from_json %({"type":"poin","x":1,"y":2}) # raises JSON::SerializableError
Point.from_json %({"type":false,"x":1,"y":2})  # raises JSON::SerializableError
Point.from_json %({"x":1,"y":2})               # raises JSON::SerializableError

constants must be a HashLiteral or a NamedTupleLiteral, mapping field names to their expected constant values. This implies every type may use multiple discriminator fields, should the need arise. These literals are chosen because they look like JSON objects.

With this, disjoint unions now work out of the box, no extra converters required:

@[JSON::Serializable::Options(constants: {type: "circle"})]
class Circle
  include JSON::Serializable
  property x : Int32
  property y : Int32
  property radius : Int32
end

alias Shape = Point | Circle

shapes = Array(Shape).from_json <<-JSON
[
  {
    "type": "point",
    "x": 2,
    "y": 8
  },
  {
    "type": "circle",
    "x": 4,
    "y": 7,
    "radius": 5
  }
]
JSON
shapes         # => [#<Point:0x7eff847a8e20 @x=2, @y=8>, #<Circle:0x7eff847abe20 @x=4, @y=7, @radius=5>]
shapes.to_json # => [{"type":"point","x":2,"y":8},{"type":"circle","x":4,"y":7,"radius":5}]

Next, we extend this behaviour to all abstract classes as well: the result of T.from_json(json) is simply the first concrete, possibly indirect, subclass U of T such that U.from_json(json) returns successfully. This is probably the only sensible interpretation for T.from_json with an abstract T. This could be done in e.g. new_from_json_pull_parser, which currently fails to compile because T.allocate does not exist; the only way to get the new behaviour is to remove any uses of use_json_discriminator in the abstract superclass T. Then we arrive at the following reimplementation of use_json_discriminator's example code:

abstract class Shape
  include JSON::Serializable
end

@[JSON::Serializable::Options(constants: {type: "point"})]
class Point < Shape; ...; end

@[JSON::Serializable::Options(constants: {type: "circle"})]
class Circle < Shape; ...; end

Shape.from_json %({"type":"point","x":2,"y":8})             # => #<Point:0x7fee792b7e40 @x=2, @y=8>
Shape.from_json %({"type":"circle","x":4,"y":7,"radius":5}) # => #<Circle:0x7fee792bae20 @x=4, @y=7, @radius=5>

Likewise, to support modules we check on all includers of T, possibly indirect ones, that are classes. Unfortunately, this isn't as straightforward because only TypeNode#all_subclasses exists, but not #all_includers. JSON::Serializable's included hook would also have to be slightly adjusted to support module includers.

In the above snippets, Shape never needs to be aware that its subtypes use a discriminator field; the types that need discriminators define them right at their own declarations. Indeed, some hierarchies do not require a discriminator at all, because the union interpretation is sufficient. The following shall be supported too:

abstract class Foo
  include JSON::Serializable
end

class Bar1 < Foo
  property x : Int32
end

class Bar2 < Foo
  property x : String
end

Array(Foo).from_json %([{"x":1},{"x":""}]) # => [#<Bar1:0x7f668d3f3e50 @x=1>, #<Bar2:0x7f668d3f2d60 @x="">]

Going in the opposite direction, this treatment now supports discriminators over multiple field names, which are sometimes seen in the wild: (we assume the discriminators are mutually exclusive, or we could include JSON::Serializable::Strict)

@[JSON::Serializable::Options(constants: {is_point: true})]
class Point < Shape; ...; end

@[JSON::Serializable::Options(constants: {is_circle: true})]
class Circle < Shape; ...; end

shape = Shape.from_json %({"is_point":true,"x":2,"y":8})
shape         # => #<Point:0x7f8a354a8e40 @x=2, @y=8>
shape.to_json # => {"is_point":true,"x":2,"y":8}

shape = Shape.from_json %({"is_circle":true,"x":4,"y":7,"radius":5})
shape         # => #<Circle:0x7f8a354abec0 @x=4, @y=7, @radius=5>
shape.to_json # => {"is_circle":true,"x":4,"y":7,"radius":5}

A proof-of-concept implementation is available here. Float and null constants are supported on top of what we have now, but Crystal constants are not ready, nor is YAML serialization. Also it might be worth pre-fetching the constant fields in new_from_json_pull_parser / new_from_yaml_node to speed up the dispatch to subtypes, instead of reading the same constants over and over again in every subtype. Note that the compiler and the standard library themselves don't use discriminators.

Related: #8473

[cyangle]

Is it possible to annotate the constants directly?

class Point
  include JSON::Serializable
  @[JSON::Constant(key: "type")]
  TYPE = "point"
  property x : Int32
  property y : Int32
end

[HertzDevil]

There is no way to access annotations on constants, as their TypeNodes are inaccessible in the macro language.

[cyangle]

Anyway, this is a brilliant idea. I hope crystal could get it soon.

[kostya]

i dont think this is good, because, in 1 project i have hundreds classes, and generate use_json_discriminator in macro automatically, but with this i need to type such things all the time.

@[JSON::Serializable::Options(constants: {type: "point"})]
class Point < Shape; ...; end

@[JSON::Serializable::Options(constants: {type: "circle"})]
class Circle < Shape; ...; end

[HertzDevil]

You can attach annotations by reopening the classes too. In particular you can reopen the current class with something like:

class Foo
  {% begin %}
    @[JSON::Serializable::Options(...)]
    class ::{{ @type }}
    end
  {% end %}
end

So if you can use a macro to generate the use_*_discriminator call, you can use a macro to generate those Options annotations.

(In fact I wonder if this issue can be implemented on top of use_*_discriminator this way)

[kostya]

i do it like this:

abstract class Bla
  macro finished
    use_json_discriminator("type", {
      {% for subclass in @type.all_subclasses %}
        {{subclass.name.split("::").last.underscore}} => {{subclass}},
      {% end %}
    })
  end
end