proto3 support

[kiennt]

Do you have any plan to support proto3?

[tomas-abrahamsson]

It might be that the already existing proto3 support in gpb is enough, but depends of course on your use case. The existing support covers proto3 general syntax and semantics, and the set of well-known types. There is also already support for the map<...,...> field type introduced in Google protobuf 3.0.0. This map type has been introduced for both proto2 and proto3, btw.

Some notable proto3 features that are missing from gpb as compared to Google protobuf:

• Automatic handling (conversion) of the Any type, but if one knows in advance what types can actually appear it might well be useful as is.
• The JSON mapping (so far, this has been low prio)

[tomas-abrahamsson]

Hello again, I've now also added possibility to plug in automatic conversion/translation of Any types (the any_translate option), so for now, I'm considering most of the proto3 support to be in place. I'm closing this issue, but you're of course welcome to re-open it if you have any further questions

[leozilla]

Are there some plans to support the JSON mapping?

Its valueable to have for a couple of reasons:

• it allows you to switch between human readable and binary based on demand (debugging)
• it allows for migration periods when an old consumer does not want to use protobuf etc but instead forces you to use JSON. You can still define your messages as protos with all the advantages but just serialize them as JSON.
• it often is a good argument for convincing ppl to move to protobuf when you can tell them: "hey we can start with JSON and switch to protobuf format when we feel confident with it"

[tomas-abrahamsson]

Are there some plans to support the JSON mapping?

At the moment, I've not got any firm plans for JSON, I've so far still considered it low prio. This is the first time I've seen expressed interest for it, if I remember correctly. In general I agree on your arguments for the use of such functionality, of course, although I don't really know what form JSON support should take.

First some updates on what's happened meanwhile: I've extended and generalized some options with eventual possibility for JSON support in mind:

• the translations mechanism (this one also had other drivers)
• the {maps_oneof,flat} option
• the {maps_key_type,binary} option could also come into play

My thinking here has been that a suitable set of translations could probably automatically convert to or from JSON on the fly during encoding/decoding.

There are a some open questions:

1. What Erlang-representation of JSON to go for?

For the moment, I've more or less assumed map() type values, with keys as either atoms or possibly binaries. I've so far only used the jsx which has two formats (maps or key-value lists with special representation of empty objects). Other JSON libs probably has smaller or larger deviations.

Related: how to represent oneof values? I've more or less assumed different keys (ie {maps_oneof,flat}, instead of the 2-tuples {field,Value} that are more common with gpb. I've assumed this in part because tuples generally don't have a natural representation in JSON, and also in part because that's how protoc's python mapping does it)

2. What are the use cases?

The normal use case of gpb is protobuf-binary⇄Erlang-value.

The various options refered above (translation etc) opens up for another use case: protobuf-binary⇄json-packable Erlang value.

But there are some other possible use cases that one could think of would, that would not easily be covered by the set of options above, but would require some new framworking:

• protobuf-binary⇄json-encoded string/binary
• json-encoded string/binary⇄Erlang-value, ie the Erlang-value on the same format as for protobuf-binary⇄Erlang-value, for example so that it would be easy to swap backends protobuf-binary vs json, to support your last argument bullet or maybe last two bullets with mostly just a flip of an option.

---

Well, this got into a long write-up about the current state of JSON support (or non-support at the moment :) ) Thoughts and feedback is of course very welcome!

[leozilla]

Thank you for your very detailed answer!

Unfortunately I dont know Erlang very well, I actually want to use protobuf in elixir which I just started to learn/use last week. I saw that the elixir protobuf library (https://github.com/bitwalker/exprotobuf) is actually using this library under the hood so I came here to ask for support.

I cannot really answer your two questions, I think I can only point you to the JSON mapping defined in https://developers.google.com/protocol-buffers/docs/proto3#json.

Do you think any of your suggestions is worth implementing? Should I create a new issue for the json support? as this issue is already closed..

[tomas-abrahamsson]

Should I create a new issue for the json support?

Yes, that may be best.

Regarding my thoughts on exactly which route to go, I think best might be to implement some kind of support one way or the other, to see if it is enough or not, and take any improvements from there.