Possible replacements for JSON communication between frontend and backend

[ravenclaw900]

There are 3 main things that I would like when communicating between the frontend and backend that JSON doesn't provide:

1. Syncing types between frontend and backend (TS/Rust) automatically
2. Validation on frontend of sent messages
3. Ability to easily encode binary data in the messages

1 and 2 can easily be provided by using some sort of schema, which I think would probably be a good idea because the dashboard's communication is completely internal, with no outside process trying to interface with it. 3 would be useful to encode file data (for instance) in actual messages instead of just sending it without any metadata. After looking, there are a number of possible solutions:

• tsync (Rust: tsync)
  • Pros
    • Just adds a proc macro to the backend, shouldn't have any effects in the final binary
    • Works with any Serde serialization library
  • Cons
    • Doesn't solve points 2 or 3
• AJV with JTD schemas (TS: ajv, Both (codegen): jtd-codegen)
  • Pros:
    • Allows keeping existing JSON code, won't bloat binary with another library
  • Cons:
    • Can't serialize u64
    • Doesn't solve point 3
• MessagePack (TS: msgpackr, Rust: rmp, rmp-serde)
  • Pros:
    • Allows keeping same serialization structure with Serde
    • Smaller and faster than JSON, while being basically a drop-in replacement
  • Cons:
    • Doesn't solve points 1 or 2 (though 1 could be solved with the addition of tsync)
• Protocol Buffers (TS: @protobuf-ts/plugin, @protobuf-ts/runtime or ts-proto, Rust: prost)
  • Pros:
    • Widely supported
    • Takes care of all 3 points
    • Fast and small format
    • Probably easiest 'Buffer' format to add to both frontend and backend
  • Cons:
    • Unlike some other formats, requires deserialization before usage
    • Might add some code bloat(?)
    • Doesn't support smaller number types (u16, for instance)
• FlatBuffers (TS: flatbuffers, Rust: flatbuffers)
  • Pros:
    • Allows reading directly from wire format
    • Official support for both Rust and TS, instead of having to use 3rd party plugins
    • Smaller code size than Protobuf
    • Takes care of all 3 points
    • Provides an easy-to-use object API for TS
  • Cons:
    • Implementation is more difficult than Protobufs, but still easier than Cap'n Proto (made even easier by the object API for TS)
    • Limited documentation
• Cap'n Proto (TS: capnp-ts, Rust: capnp)
  • Pros:
    • Allows reading directly from wire format
    • Takes care of all 3 points
    • Easy to read and write schema format
  • Cons:
    • Complicated to implement for both frontend and backend
    • capnpc-ts (typescript complier) seems to be broken
    • Larger wire format, though can be packed down

At first glance, FlatBuffers look like the way to go for our needs, but I'll have to do some more testing before making a decision.

[MichaIng]

Automatic message validation seems reasonable and easier/goes along with syncing types.

Could you give me a hint where we transfer binary data (within our code scope)?

[ravenclaw900]

There are 3 places specifically, all of which currently use different websocket connections from the main one to allow transferring raw binary data. Uploading and downloading files/zip-encoded folders use /ws/file, which currently only sends the file data, with no specific message wrapper. The terminal uses /ws/term, but that probably won't be helped much by a binary encoding format, as it's connected directly to xterm.js in the frontend. In addition, using a binary encoding format often encodes faster and provides smaller final encodings compared to a text encoding like JSON.

[MichaIng]

Ah right, upload/download files of course. What benefit of a message wrapper do you think of?

For the terminal I thought so that this is all handled in the xterm module so that we cannot hook well inside, respectively it won't benefit.

In addition, using a binary encoding format often encodes faster and provides smaller final encodings compared to a text encoding like JSON.

If there is no additional conversion required when generating binary data from information respectively parsing binary data vs JSON, then I agree. I just remember from some Python coding where transfer modules require binary data at some point and it added an additional conversion step on both ends as e.g. first a JSON string is (manually) generated and then needs to be binary-encoded, and at the other end first decoded before the information is accessible. So there I thought that sending the only minimally larger data in plain text would be easier. But if e.g. the binary format is generated directly, instead of first a plain text JSON, then of course it doesn't matter.

[ravenclaw900]

No, there is no need to convert between JSON and the binary format, we could convert directly to it from the Rust structs (probably faster than we could convert to JSON, actually). Using a wrapper format would let us remove the separate websocket for uploading and downloading files, as the raw file data could be wrapped by the binary format and parsed the same way as everything else. In addition (I just thought of this), adding a binary wrapper could let us add checksums to the file data as it's being sent, to make sure that it isn't being corrupted.