Justifications for not using existing MessagePack / BSON, etc.

[ioquatix]

I think your section about alternatives is open to interpretation and maybe the reasoning can be improved.

The first and most important is the fact that such serialization protocols are not specifically designed for a request-response server-client chat, so clients and servers would be required to agree on an additional protocol on top of the underlying serialization protocol.

At least with MessagePack, you can stream binary messages very easily. So, request/response is pretty trivial to implement.

A different problem is the fact that such serialization protocols are more complex than RESP, so client libraries would have to rely on a separated library implementing the serialization protocol.

In my mind, this is both a pro and a con. Obviously, the reason you mention makes sense. However, in every library I've seen, some kind of protocol parser must be implemented. Honestly, I'd rather just use a 3rd party implementation like MessagePack which is proven to be fast/memory efficient, than have every library roll their own.

Finally certain features like transferring large strings with initially unknown length, or streaming of arrays, two features that RESP3 supports and that this specification describes, are not easy to found among existing serialization protocols.

While this might be more tricky to implement, there is certainly no reason why you can't split such replies up into continuation messages. Given that you want to have features like client key cache eviction, streaming huge amounts of data is often a bad idea. It also introduces memory usage issues as client has no choice but to read all data to reconstruct the reply, rather than handling discrete chunks.

Not saying I don't agree with the points, I just think they don't present a clear cut reason not to use MessagePack, BSON, etc.

Honestly, from my POV, I wish you just used some existing format, rather than inventing your own. Even if you needed to add some semantics on top of it, it would be easier for client implementations IMHO. So, best to make a really strong case for RESP3.

[ioquatix]

Just to add, a stronger argument IMHO, is that RESP3 is backwards compatible with RESP2, and that all the existing justifications for RESP2 continue to exist, e.g. human readable, etc.

[antirez]

@ioquatix I didn't mean that MessagePack can't be used as request-response, but you have to invent a protocol over it to do so: it is not at the level of abstraction of RESP3 being a serialization protocol, so you have to decide, for instance, that the replies provided by MessagePack have a first element that represents the reply type and so forth.

About your parsing libraries point: we have seen terrible MessagePack implementations. We use MP in Redis Word as a serialization format, for instance it is supported by Lua, and I had to write an implementation of MP from scratch because the alternatives were too por. RESP simplicity means the parsing is kinda of a side effect of implementing the client itself, since it's a tiny part of the whole client and is very simple and non binary encoded.

Re continuation messages: yep, that's true, and I think RESP3 will end using such encoding. However it is not always what you want, we had to debate if instead, in order to avoid any buffering, we were more happy with a different setup, that is, using an EOF marker. In such case MP would be not ideal at all.

I truly believe that instead relying in a more complex serialization format would make the client implementations more complex: there are libraries, but most code in the world is terrible, and also creates a new dependency between the library and the external world.

Thanks for the reasoning!

[ioquatix]

Yeah all makes sense.