Does ocaml-protoc support text-mode serialization?

[chetmurthy]

With protobufs, one of the really nice things, is that you can use binary-mode serialization (for efficiency) or text-mode for debugging, passing config args, etc, etc, etc. Basically, who needs YAML, when you can use text-mode protos, eh?

Does ocaml-protoc support this?

[mransan]

It supports serializing to a string in a text format, however it does not support deserializing from that format. @chetmurthy were you thinking of a specific text format with some clear rules .. something similar to json encoding in proto3 (Which is something i would like to add support for)

[chetmurthy]

@mransan I was thinking specifically of the Google protobuf text-mode format -- the one that's supported by protobufs out-of-the-box in C/C++/other-langs.

When I worked at Google, it was quite common to express configs in this text-mode format, and I'd like to do the same, now that I no longer do.

[chetmurthy]

I should have added that this format was an "official" wire-format. So if you wanted, you could have run an RPC system, with the payloads (not the framing, obv) in cleartext, using this. So it would need to both serialize and deserialize -- just like the other (binary) official wire-format.

[mransan]

I was not aware of this format and i think it's definitely needed to have a text format for use case that requires human readable output.

However i could not find any specification for this text encoding so it would need to be reverse engineered a bit. What do you think of the JSon encoding in proto3, the advantage is that the rules are fully specified (including field attributes to customize the output). The other advantage of JSon would be it's easier integration with a JavaScript front end for instance.

The drawback is that JSon is not the best format for configuration file use case since it's lacking comments.

[chetmurthy]

Can you point me at some examples of the JSON format on nontrivial data? I ask b/c for sure, the text-format is really, really nice for writing out nontrivial data-structures (for configuration, but also for tests -- you can parse up a big-ass protobuf message nicely from a raw-text string ....

I'll look around for documentation on the text-format.

[chetmurthy]

Oh, here's a link with some examples:

http://stackoverflow.com/questions/18873924/what-does-the-protobuf-text-format-look-like

E.g. a message

message Pair { required string key = 1; optional string value = 2; }

serializes like

pairs { key: "sdfff" value: "q\"qq\q\n" } pairs { key: " sdfff2 \321\202\320\265\321\201\321\202 " value: "q\tqq<>q2&\001\377" }

[mransan]

Thanks for the example. I also see need to try more the JSon serialization and see how it compares.

[chetmurthy]

Well, one thing I wonder is: why not just try to "extend" protobuf3 to support ocaml directly?

just wondering ....

Not saying that you should do it, but rather, wondering why you didn't go down that path.

[mransan]

Do you mean providing a protoc plugin rather than a whole compiler ?

If so then yes i still consider it, I think it could be a good incentive for developer familiar with the tool to try out OCaml in general. For OCaml users depending on protoc is an added dependency if you only care about OCaml code.

In order to implement a plugin you need to already be able to deserialize protobuf messages since the protocol between the front end protoc and the language specific backend is via a protobuf message.

Now that we have ocaml-protoc we could generate the code for the descriptor.proto and essentially boot strap the plugin. Most of the code of ocaml-protoc can then be re-used.

[chetmurthy]

[Let's see if I can reply directly from my mail-reader.]

Excellent! OK, I'm -buried- in a completely unrelated project (for which I had -hoped- I could use text-mode protobufs as my config format). But as soon as I finish it, I'll contact you, and maybe I'll write the text-mode protobufs?

--chet--

On Wednesday, 4 May 2016 20:18:17 PDT mransan wrote:

Do you mean providing a protoc plugin rather than a whole compiler ?

If so then yes i still consider it, I think it could be a good incentive for developer familiar with the tool to try out OCaml in general. For OCaml users depending on protoc is an added dependency if you only care about OCaml code.

In order to implement a plugin you need to already be able to deserialize protobuf messages since the protocol between the front end protoc and the language specific backend is via a protobuf message.

Now that we have ocaml-protoc we could generate the code for the descriptor.proto and essentially boot strap the plugin. Most of the code of ocaml-protoc can then be re-used.

---

You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/mransan/ocaml-protoc/issues/79#issuecomment-217068370

[mransan]

Sounds good! Good luck with your project!

[mransan]

JSON is human friendly enough for now.