search

omnistreams / omnistreams-js

0 stars 0 forks source link

Potential problems with large messages #3

Open anderspitman opened 3 weeks ago

anderspitman commented 3 weeks ago

omnistreams aims to support passing arbitrary messages in a descrete send. However, at one point I ran into problems with Chrome trying to send chunks that were too big, which resulted in this commit: https://github.com/omnistreams/omnistreams-js/commit/3d5fff07450fdd888998cfb67c3e781330060275

What that means is we're currently limited to messages that are 64KiB. Need to do more exploration now that things are more stable and see if we can better understand the limitations. I suspect there may have been confounding variables with the Golang implementation and maybe the window size as well.

Worst case we might need to keep the muxado frame length and permit omnistreams frames to be split across multiple WebSocket frames. It would be nice to avoid that, but it's not too completely awful to implement. Here's the current implementation which is only needed if interfacing with a muxado peer: https://github.com/omnistreams/omnistreams-js/blob/4de2b77d3375a6e4059b5725f6df40989414e00c/index.js#L68

anderspitman commented 3 weeks ago

Just did a quick test and when piping a file into a TransformStream, Chrome (127.0.6533.99) is yielding chunks of size 2097152 out the other side, whereas Firefox (129.0) is only yielding size 65536 chunks. 2MiB might be a little big. It's tricky. Whatever value we accept, the default window size needs to be at least that big. Maybe we just punt it to devs: make the window size configurable (with a reasonable default), and make the max chunk size always match the window size.