Data format for mutinyd unix socket API

[caolan]

I'd prefer if it did not require a schema and code generate (e.g. protobuf). This allows third party clients (perhaps a tray icon for your desktop of choice, for example). to evolve at a different pace if we take some care with the API. We're unlikely to need the added compression of a schema-based format since this will be local IPC and we're only supporting a single user.

I also want the format to support binary data natively in case we need to efficiently transfer files, public keys, hashes, etc.

Candidates

• MessagePack - widely used self-describing binary data format
• BSON - supports binary data but more complicated than we need and lacks a proper standard.
• Bencode - used by BitTorrent, really simple format providing byte strings, integers, lists, dictionaries. I'm a long time fan, especially for languages that lack a JSON parser because it's easy to knock up an encode/decoder. Perhaps a little too basic.
• Smile - looks like a reasonable binary data format. Based on JSON and created by the developers of the Jackson library for Java. Supports JSON data type + binary data. Fewer extensions than BSON. The ability to use back references for keys for better compression is interesting. Does not appear to be widely adopted.
• UBJSON - seems like a more sane BSON
• CBOR - influenced by MessagePack. Similar aims to UBJSON but it appears CBOR is the more widely adopted. Spec with IETF and used in WebAuthn standard from W3C. Seems to achieve predictable best and worst case compression. Support for optional extensions via 'tags'.

I'm leaning towards CBOR, but MessagePack has the advantage of having an encode/decoder in the Deno standard library.

[caolan]

It might make sense to have a simple wrapper that encodes a result code and content length then have the content depend on the method call and result code. E.g. JSON or raw bytes of a file. That would make it easy to do sendfile on linux for large files.

[caolan]

Most (all?) of these formats would make it possible to decode a stream of values without extra envelopes or delimiters. However, a simple length integer followed by the encoded value would make it easier to recover from a parse failure. If a message of a known length fails to parse it might be possible to recover by reading the rest of the content until the length is removed from the stream then begin parsing again. Knowing the length ahead of time also makes it possible to allocate the full buffer required in advance and do zero-copy deserialisation of certain values.

Alternatively, it might just make sense to close the connection and reopen it to recover.

Note that a preceding length integer precludes streaming responses.

[caolan]

The data format of the mutinyd will be:

<length><message>

• <length> - An unsigned 32-bit integer in network byte order (big endian)
• <message> - A MessagePack encoded value of size <length>

If we require larger streaming values, e.g. sending a big file, the caller will be pointed to where to find the data rather than including it directly in the mutinyd request/response.