Possibly switch to flatbuffers?

[mborgerson]

Just came across this. Tbh don't know much about it except it allows reads w/o deserialization. Recording here for later research.

https://github.com/google/flatbuffers

[JayFoxRox]

I assume this is to replace protobuf. If so, it still sounds like overkill: Why add a dep and the burden of learning a new API, for a protocol this simple?

I still think that python struct.pack or a simple packed C struct is enough for our protocol. We have like 10 commands - and we hardly need more than that. Creating raw binary data in every language is very easy. We just need a Protocol.md or something to keep track of them.

The only tricky thing would be compressed readback for larger data reads. But most existing solutions don't solve this properly either (compressing / adding latency when we don't want and vice versa). We should have finer control over such things as we likely need it to stream a lot of data quickly, so we need to fine tune the compression settings. I think gzip might be a good choice for this - which is also widely available, very standard, and could be an optional feature in our own protocol.

Desynchronization (where the parser might be halfway into a packet) with the Xbox shouldn't be an issue either as we likely won't have packloss. Even if we do, there's probably a couple simple methods to avoid it. But I don't think we'll need this.

[mborgerson]

The motivation for using something like protobuf (or possible flatbuffers?) is that we don't need to maintain two sets of interfaces, just one schema file, and we don't have to write any [de-]serialization code--we just let the library code handle it. In this case, for RDT, you're not wrong--the protocol is very simple and we could just pack it ourselves. Using a struct and Python's ctypes would make this easy--but we need to be careful about avoiding pointers (char *).

For compression, I'd probably prefer to use LZ4. It's embeddable, lightweight, and produces acceptable compression ratios at really high speeds.