Feature request/idea: Protobuf compatibility?

[KeithHenry]

I really like this idea - it seems like a really good way to use ArrayBuffer.

However, while this really helps with large objects passed between workers, the service is still sending JSON serialised content, leaving a process something like:

service > send big JSON > web worker > sync parse JSON > do stuff > define BBO > postMessage ArrayBuffer > UI JS use BBO.

From the server we have the option of using https://github.com/protocolbuffers/protobuf to drastically reduce the serialisation size, but there's currently no easy way for JS to parse the protobuf messages, and even if there was we have:

service > protobuf > web worker > do stuff > parse protobuf > define BBO > postMessage ArrayBuffer > UI JS use BBO.

It seems like a protobuf message might be a good candidate for the structure behind the BufferBackedObject, as it is just a BLOB, and every step could pass the same binary array using the same wrapper:

service > protobuf > web worker > do stuff with BBO wrapper > postMessage protobuf > UI JS use BBO wrapper.

Protobuf uses .proto file to define the schema - ideally we'd either be looking at a way to use those, either ProtobufBackedObject('proto file content/definition') or some kind of build tool to turn .proto files into the matching ProtobufBackedObject JS descriptors.

[surma]

I don’t think I fully understand. Protobufs are serialization schema, just like this library. You can just send ArrayBuffers using this library to virtually skip the JSON serialization/deserialization.

[KeithHenry]

@surma I might be confused on what this does...

The server is sending JSON, but could use a cross-platform binary serialisation format like Protobuf instead. I can't really use this format in my server unless it is running Node/Deno/JS or I reimplement the same serialisation format in whatever server-side tech stack.

I think BufferBackedObject gives me a really nice way to postMessage binary data between web workers and the main page JS, but it doesn't allow me to use the same binary schema in a fetch. That means two binary schemas with some kind of translation between the two (fetch protobuf/server-format binary data, deserialise to JS object, build BBO round new ArrayBuffer, copy values across, postMessage that).

I don't really want to have two independent, similar, binary serialisation definitions - I guess I'm looking for a way to avoid translating between two different binary serialisation schemas.

Am I missing something?

[surma]

Ah you didn’t mention server/client communication in your original post.

This library was written for worker communication primarily. That being said, even for server/client you can skip the JSON step:

const response = fetch("/data-endpoint").then(r => r.arrayBuffer());
const obj = new BufferBackedObject(response, { /* ... */ });

Does this answer your question/concern?

[KeithHenry]

@surma yes, thanks, something like that is just what I want to do, but that only works if the service agrees on the definition for the BufferBackedObject. Like "/data-endpoint" needs to have the same binary serialisation schema - I could do that if I'm running JS on the server (using another BBO with the same definition), but if I'm not I don't have a cross-platform way to configure it.

So ideally I'd want to have the client and server use the same binary definitions - they could both use shared descriptors somehow or have a common format that could be used to generate both.

I suggested Protobuf as it gives me that, along with a load of established server side libraries and version management (so I can add/remove fields and the binary messages still parsed by the old client - something web-worker interop doesn't really need but client-server does) but there are loads of alternatives.

It's not a concern as such, it's just that being able to fetch this format too would be a big plus :-)

[surma]

I’m sorry, but I still don’t quite follow.

Like "/data-endpoint" needs to have the same binary serialisation schema

That is always the case tho. If I added support for the protobuf serializion scheme, we agreed to use protobufs on that endpoint.

The protobuf project already allows you to share the descriptors on both the server, the main thread and workers side. So this library doesn’t add any value in that scenario. Just use protobufs then. I was mostly looking to write something lightweight and simple that doesn’t require a compilation step.

[KeithHenry]

@surma fair enough :-) lightweight and simple (and the elegant DataView wrapper approach rather than deserialise/reserialise) is kind of what I'm trying to figure out, but I think I'm wasting your time here, sorry. Thanks for taking the time to put me straight.