JSON mapping support

[leozilla]

As already discussed in this issue: https://github.com/tomas-abrahamsson/gpb/issues/61

Support for the JSON mapping as described here (https://developers.google.com/protocol-buffers/docs/proto3#json) would allow to switch between human readable and the Protobuf machine optimized data formats on a per use case basis.

[tomas-abrahamsson]

Yep. I will look into it, but don't hold your breath.

[DanielUranga]

👍 How hard would it be to implement? What could I do to help it become a reality?

[DanielUranga]

I answer myself: https://github.com/tomas-abrahamsson/gpb/issues/61#issuecomment-471120636

[leozilla]

@tomas-abrahamsson any updates? just so you know, once u implemented this I will use it as a base for the same impl in elixir. So we would catch 2 birds with one stone (I hope u dont like birds.. lol)

[tomas-abrahamsson]

No updates yet, unfortunately. I've had little time for this for quite a while now, but I hope to be able to tend to it soon again. It is the next big thing on the agenda for gpb. Sorry, and thanks for having patience.

A while ago I took a look and realized that the translations approach, that I talked about earlier, was probably not a good idea. Better to generate specialized encoders to and decoders from json. I also took a look at some json libraries, and it seems it would not be too difficult to make it possible to have a generation-time option for supporting jsx, jiffy and mochijson2 (and maps). So in a way, I've at least set a direction.

[tomas-abrahamsson]

I just pushed some work-in-progress to a json branch, if you want to play with it. Note that the branch may get force-pushed in the future. If the json option is specified, the generated code will contain a to_json/1,2,3 function. So far, I've only done encoding, not decoding, so there is not yet any from_json function. That's next step.

My idea is that the generated code will output a JSON representation that can be used directly with some common JSON encoder/decoder libraries to produce/parse JSON text. Ie the code generated from gpb will not do the actual JSON text generation/parsing itself. I hope this approach is ok. There are presets to generate code with format for jsx, mochijson2, jiffy and map, but it is also possible to control details, for interwork with other JSON libraries. The list of new options are:

• json top-level option to generate the to_json function or not
• json_emit_type_defaults to always emit json key-value items also when the value is the type's default. Normally, it is not included in the json if the value is the type's default, as per the language guide. (For now, this is a generation-time option and not a runtime option)
• {json_format, jsx | mochijson2 | jiffy | map} -- will expand to the corresponding below options for object, key, array and string.
• {json_object_format,Format}
  • eep18 -- empty objects are [{}] and non-empty objects are proplists (eg the jsx library) (default if the maps option is not specified, ie if generating for records)
  • {proplist} -- the proplist is wrapped in a tuple (eg the jiffy library)
  • {atom(), proplist} the proplist is wrapped in a tagged tuple, and it is possible to specify the tag (eg the mochijson2 library)
  • map -- use maps (default if the maps option is specified) (several of the libraries)
• {json_key_format,Format}
  • atom
  • binary (default)
  • string
• {json_array_format,Format}
  • list (default)
  • {atom(), list} -- a list in a tagged tuple, with possibility to specify the tag
• {json_string_format,Format}
  • list
  • binary (default)
• {json_null, atom()} -- to specify the JSON null value. Default is null.

I have not yet double checked super-much if the output works well with all targeted JSON libraries. I also have not yet cross-checked with Google's protobuf's JSON, I've only tried to follow what the language guide says.

There is not yet support for the json_name proto option.

I've been thinking whether or not to keep the top-level json switch/option. I'm thinking now as things are a bit experimental still, it may be good to have it. Later on it may default to true, or go away entirely.

[tomas-abrahamsson]

I pushed again to the json branch (force push). It now generates to_json and from_json. I've interop-tested it with the json libs, and also with JSON from Google's protobuf, but there may of course be bugs. You're welcome to take it for a spin, please let me know if you think it looks useful, also from an Elixir perspective.

Except for the addition of a generated from_json function, compared to earlier versions of the branch, there are just a few additional user-visible changes:

• the json_emit_type_defaults option changed name to json_always_print_primitive_fields
• new option json_preserve_proto_field_names

The reason for these changes is I think it better harmonizes with the names in the JsonPrintOptions of Google's protobuf.

[tomas-abrahamsson]

FYI: I just force pushed again. No changes to JSON functionality. (Changes are: eunit test refactorizations for NIF tests and robustification to locating vendored proto3 well-knowns.)

[tomas-abrahamsson]

FYI again: Pushed again, I've been doing some probing of Google's protobuf for decoding JSON, ie the from_json function, to see what it accepts that's just outside of the specified and documented values, to try to accept the same, for better interop:

• For bool, it turns out Google's protobuf also accepts the strings "true" and "false" (case insensitively)
• Google's protobuf from 3.8.0 (currently rc1) no longer accepts enum strings case insensitively by default but there's an option to do so. So I added a corresponding option to gpb to be able to accept enum strings case insensitively: json_case_insensitive_enum_parsing
• Parsing of floating point value strings is now a little bit more liberal and accepts some other common representations just outside the json spec (Google's protobuf accepts these too)

@leozilla Have you had any chance yet to take a look? If things looks generally ok, there's always also the route available to merge it and fix any more or less minor changes later on as we go along.

[tomas-abrahamsson]

Hi, I decided to merge this, so it is now included in 4.9.0, which I just pushed. Should you discover anything that would need reworking or fixups, please feel free to open an issue for it.