Use different serialization format for P2P messages.

[artemii235]

JSON is too "fat" for such purpose, e.g. notify message

{
        "method": "notify",
        "rmd160": "9562c4033b6ac1ea2378636a782ce5fdf7ee9a2d",
        "pub": "5eb48483573d44f1b24e33414273384c2f0ae15ecab7f700fb3042f904b09820",
        "pubsecp": "0342407c81e408d9d6cdec35576d7284b712ee4062cb908574b5bc6bb46406f8ad",
        "timestamp": 1541434098,
        "sig":  "1f1e2198d890eeb2fc0004d092ff1266c1be10ca16a0cbe169652c2dc1b3150e5918fd9c7fc5161a8f05f4384eb05fc92e4e9c1abb551795f447b0433954f29990",
        "isLP": "45.32.19.196",
        "session": 1540419658,
    }

contains rmd160, pub, pubsecp, sig fields which are bytes sequence encoded in hex - it makes the message almost 2x larger not providing any real benefit. So switching to some binary format might help us to decrease the message sizes significantly which will be the most important for mobile users. The requirements are:

1. Small resulting payload size.
2. Easy backward and forward compatibility.

As of now I see following options:

1. Use our own implementation maybe based on https://github.com/paritytech/parity-bitcoin/tree/master/serialization
   • pros
   • small and efficient.
   • we will have full control of serialization lib, easy to extend by ourselves.
   • cons
   • Extension will require some effort from us
   • The data structures won't be cross-compatible with other languages, we will need to reinvent the wheel every time (e.g. for browser-only client or if someone would like to create MM2 client on different language).
2. Google protocol buffers
   • pros
   • claims to be simple and fast.
   • Various Rust implementations.
   • Cross-compatible with a lot of languages.
   • Used by libp2p which we might support in the future.
   • cons
   • Rust implementations are not official so there may be some delays in updates (maybe it's not critical).
   • Requires additional tooling to compile proto structs to Rust, e.g. libp2p is full of such scripts
3. Something else?

[ArtemGr]

A couple of extra options to the mix:

We're using a bit of https://en.wikipedia.org/wiki/Bencode with libtorrent and I like the compactness and the nice property of it having a single binary representation (the bijection property), useful for duplicate detection and caching. Though it's a bit more restrictive than JSON.

For simple stuff there are also netstrings, e.g. the example above might be encoded as a list of key-value pairs: 6:method,6:notify,6:rmd160,11:...,3:pub:,32:...,

I should also mention zstandard compression which we're now using with HTTP fallback. It supports a static dictionary, allowing us to remove the overhead of keys (like the "method", "notify", "rmd" and "pub" keys above). zstandard allows us to use JSON (to which CRDTs in HTTP fallback are limited) with a close-to-binary size efficiency. P.S. Though zstandard isn't readily portable.

[sergeyboyko0791]

I think we should consider switching from rmp_serde to either another binary protocol like protobuf or some human-readable text protocol like Json. It becomes hard to add new fields to messages, and sometimes it requires the use of crutches like this https://github.com/KomodoPlatform/atomicDEX-API/issues/1487

cc: @artemii235