Code generation for messages

[j0sh]

Currently we use a mix of encodings: JSON to wrap Golang gob-encoded messages.

• The gob encoding is not portable. While theoretically documented in https://golang.org/pkg/encoding/gob/ , in practice gob encoding restricts usage to Go-based clients only. We should use a more portable serialization format so nodes can be developed in other languages.
• Consider whether we should keep the current JSON wrapper: with a typed, schema-based serialization (capnp, protobufs), we can rely more on code generation, and do less manual handling for each message, reducing the amount of boilerplate (eg, data.go).
• Fine grained versioning built into the networking protocol, assisted by the protocol compiler. https://capnproto.org/language.html#evolving-your-protocol

[jozanza]

I haven't used them, but FlatBuffers seems like the best option for optimizing the amount of unpacking/parsing involved (best performance).

Otherwise, I'd maybe stick with JSON during a prototyping/beta phase. It would be the fastest way to iterate on message types initially. Then we could always move to flatbuffers or anything else once our message schemas become more stable.

Comparison from Cap'n Proto website for reference

[j0sh]

Oh very cool, I hadn't heard of Flat Buffers. Seems to take a very similar approach as Capn'Proto -- it essentially write()s the in-memory struct on the wire. Golang would still have a serialization step to accommodate the internal representation. I think I still have a slight preference for Capn'Proto, since more languages are supported and it has Wireshark dissectors available (very handy!) The project/tooling also seems more... decentralized...? which is a plus.

Updating a schema is as fast as editing the definition. The code is regenerated at build time in a way we can be certain is correct. It's really nice to be able to read/write directly from typed structs in a safe manner.

JSON seems like it'd still require manual munging, especially for binary data. Dynamic access to fields also tends to blur the separation between the message definition and the handling of that message, which ultimately makes the code less clear, in my experience. Open to changing my mind though.

[jozanza]

Yeah it's much easier passing JSON around in JavaScript at least...not sure about other languages haha.

If we are actually sending binary data (not just JSON types), then I'd be inclined to use something like FlatBuffers immediately. Also, I just came across an interesting featuer of FlatBuffers in this thread

[ericxtang]

Ahh yeah - json + gob is definitely not the best approach... I was planning to eventually use protobuf (suggested by the ipfs team and I think you can find code examples for), but open to better solutions.

[binary132]

@j0sh , I don't think the Go implementation of flatbuffer requires a deserialization step, and go generate is sufficient to trigger the flatbuffer schema compilation step.

I'm curious to see where you got the idea that Go FB requires deserialization, since that would also impact my usecase.