tomas-abrahamsson / gpb

A Google Protobuf implementation for Erlang
Other
557 stars 153 forks source link

proto3 support #61

Closed kiennt closed 8 years ago

kiennt commented 8 years ago

Do you have any plan to support proto3?

tomas-abrahamsson commented 8 years ago

It might be that the already existing proto3 support in gpb is enough, but depends of course on your use case. The existing support covers proto3 general syntax and semantics, and the set of well-known types. There is also already support for the map<...,...> field type introduced in Google protobuf 3.0.0. This map type has been introduced for both proto2 and proto3, btw.

Some notable proto3 features that are missing from gpb as compared to Google protobuf:

tomas-abrahamsson commented 8 years ago

Hello again, I've now also added possibility to plug in automatic conversion/translation of Any types (the any_translate option), so for now, I'm considering most of the proto3 support to be in place. I'm closing this issue, but you're of course welcome to re-open it if you have any further questions

leozilla commented 5 years ago

Are there some plans to support the JSON mapping?

Its valueable to have for a couple of reasons:

tomas-abrahamsson commented 5 years ago

Are there some plans to support the JSON mapping?

At the moment, I've not got any firm plans for JSON, I've so far still considered it low prio. This is the first time I've seen expressed interest for it, if I remember correctly. In general I agree on your arguments for the use of such functionality, of course, although I don't really know what form JSON support should take.

First some updates on what's happened meanwhile: I've extended and generalized some options with eventual possibility for JSON support in mind:

My thinking here has been that a suitable set of translations could probably automatically convert to or from JSON on the fly during encoding/decoding.

There are a some open questions:

1. What Erlang-representation of JSON to go for?

For the moment, I've more or less assumed map() type values, with keys as either atoms or possibly binaries. I've so far only used the jsx which has two formats (maps or key-value lists with special representation of empty objects). Other JSON libs probably has smaller or larger deviations.

Related: how to represent oneof values? I've more or less assumed different keys (ie {maps_oneof,flat}, instead of the 2-tuples {field,Value} that are more common with gpb. I've assumed this in part because tuples generally don't have a natural representation in JSON, and also in part because that's how protoc's python mapping does it)

2. What are the use cases?

The normal use case of gpb is protobuf-binary⇄Erlang-value.

The various options refered above (translation etc) opens up for another use case: protobuf-binary⇄json-packable Erlang value.

But there are some other possible use cases that one could think of would, that would not easily be covered by the set of options above, but would require some new framworking:


Well, this got into a long write-up about the current state of JSON support (or non-support at the moment :) ) Thoughts and feedback is of course very welcome!

leozilla commented 5 years ago

Thank you for your very detailed answer!

Unfortunately I dont know Erlang very well, I actually want to use protobuf in elixir which I just started to learn/use last week. I saw that the elixir protobuf library (https://github.com/bitwalker/exprotobuf) is actually using this library under the hood so I came here to ask for support.

I cannot really answer your two questions, I think I can only point you to the JSON mapping defined in https://developers.google.com/protocol-buffers/docs/proto3#json.

Do you think any of your suggestions is worth implementing? Should I create a new issue for the json support? as this issue is already closed..

tomas-abrahamsson commented 5 years ago

Should I create a new issue for the json support?

Yes, that may be best.

Regarding my thoughts on exactly which route to go, I think best might be to implement some kind of support one way or the other, to see if it is enough or not, and take any improvements from there.