Discussion: Encoding and serializiation

BearGuy commented 3 years ago

Hey folks,

There has been some internal discussion about serialization and encoding, so I thought it would make sense to document some of the ideas in these discussions more publicly.

Rust gives quite a few methods to implement serialization, and currently there's a short list of the ones that have been discussed, each containing their own sets of pros and cons.

Serde/bincode

Pros

Compact, fast byte encoding native to Rust and its type system
Supported by other projects, including google/tarpc a Rust implementation of RPC that requires no additional tooling or languages to create a message schema.
Easy to implement, plug and play with Serde using the bincode crate.
Cons
Rust specific and tightly coupled, meaning almost no other language support
Less control in encoding, potentially meaning less performant

Protocol buffers

Pros

Platform and language neutral
Easier to implement than building a custom serializer
Cons
Additional complexity to manage a separate language
Less efficient than custom serializer

Serde/Custom Serializer

Pros

Complete control over encoding our structures, easy to optimize where we need it
Cons
Cost of development time, maintenance, and bug fixing
Would require recreating it for each new implementation in another language

There's a few other open-ended questions to factor into this conversation that will influence the decision. One of them in my mind is will the Rust version be the only version that is allowed to run consensus? If so, being as tightly coupled to Rust isn't as big of an issue.

Personally, at time of writing I'm leaning most towards Protocol buffers. It's a good compromise of dev complexity, language agnosticism, and efficiency that gives us decent flexibility moving forward.

If there are other standards that should be considered, please mention them! Now is the time to lay it out before anything gets implemented.

escapedcat commented 3 years ago

Tagging @batzor here for maybe more feedback on this issue in general

Bincode

I'm a big fan of using default libs to do stuff. But yesterday I tried finding anything JS-library-ish that could handle Rust-bincode. Just to see if there's something in that direction. Didn't find anything that looked hopeful.

One of them in my mind is will the Rust version be the only version that is allowed to run consensus?

I could imagine that locking in now to this might be an issue later. Especially if there's no bincode support in other languages (But maybe there's chance this will be added in the future?).
How to send/receive data with a npm-module later on?

Related bincode-repo issues:

Protocol buffers

For the lack of alternatives this sounds most reasonable

Related projects:

Custom

I am not a big fan of implementing such things in a custom way, mostly because of the mentioned cons

clayrab commented 3 years ago

One big downside of protobufs is that the entire protobuf spec becomes part of our spec. The upside is we might get considerable performance benefits.

I think bincode isn't a viable long-term solution.

"Would require recreating it for each new implementation in another language". I view this as a "pro" for custom serializers, not a con. The encoding scheme must be implementable in arbitrary languages.

batzor commented 3 years ago

I will just leave it here. Benchmarking code of protobuf and bincode. https://gist.github.com/ilblackdragon/88ab377bf0da7ff721af80cb8a11fe28

BearGuy commented 3 years ago

Just though I'd mention, benchmarking and tests of these different solutions exist in the serialization branch here in this repo for anyone who's curious or wishes to help us extend testing.

escapedcat commented 3 years ago

Linking the mentioned branches:

escapedcat commented 3 years ago

Can this be closed?

trevelyan commented 3 years ago

Thank you for the discussion everyone!

We are implementing custom serialization at this point.

SaitoTech / saito-rust