Question: message boundaries support

[OEvgeny]

Hi, First, thank you a lot for bpmux. Both are great: the idea and the implementation!

Now a bit of context on the question. My usecase involves node streams in objectMode where objects are serialized to JSON and then passed down to multiplexed streams (via BPMux). All goes well until it has to pass down a buffer longer then 16384 bytes. Buffers above this length are getting split into parts and receiver gets the message part by part. Obviously it breaks my naive implemented (de)serialization, which could be fixed i.e. I need to prefix messages with their lengths by myself. While investigating this, I found next things:

• This library uses frame-stream under the hood, which does length prefixing and should keep message boundaries (at least if I got it right).
• frame-stream is able to handle messages which are less then 2^32 in length (more than 16384 bytes).
• Almost all tests I've seen verify concatenated buffers. So, no checks for single messages are there.
• Input stream created by BPMux is unbuffered frame-stream, which seems to be right.

So, the question: Does bpmux support keeping message boundaries for such cases?

Thanks!

[davedoesdev]

Hi @OEvgeny thanks for using bpmux.

By design, bpmux does not support keeping message boundaries. It tries to be fair by splitting available space on the underlying stream equally between each multiplexed stream. At the receiving side, it makes data available to each multiplexed stream immediately when it comes in.

It would be possible to alter this behaviour but it would probably be just as easy to use your own frame-streams on the multiplexed streams to do the buffering. Would that work?

[OEvgeny]

it would probably be just as easy to use your own frame-streams on the multiplexed streams

Just tried this out. Looks like it wont work for some reason, maybe because bpmux already uses it (I'm not sure frame-stream could be used together with frame-steamed stream, although I may be wrong).

By design, bpmux does not support keeping message boundaries.

Got it! It uses frame-stream to keep boundaries for itself not multiplexed streams, as far as I can understand.

Thanks, I think this is answered, so closing this.

[davedoesdev]

@OEvgeny it should be possible to use frame-streams on the multiplexed streams. I'm due to update the deps on bpmux soon. I'll add a test for it when I do that.