search

michelle / reliable

Slow but reliable transport for unreliable DataChannels.
30 stars 19 forks source link

Binary data gets garbaged #11

Open piranna opened 11 years ago

piranna commented 11 years ago

When trying to send an image of just some KB, the content gets converted to a sequence of escaped Unicode glyphs (\"u1234") getting a transfered size sometimes more than double bigger, and worst than that, message gets truncated because of the extra size. For text files (for example HTML) there's not this problem.

piranna commented 11 years ago

Enabling the UTF8 flag on BinaryPack constructor this seems to be fixed.

michelle commented 11 years ago

I actually cannot reproduce this. Are you converting your image to a string first?

piranna commented 11 years ago

Yes, I'm moving JSON strings. El 03/05/2013 06:47, "Michelle Bu" notifications@github.com escribió:

I actually cannot reproduce this. Are you converting your image to a string first?

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17378465 .

michelle commented 11 years ago

Why not just directly send JSON objects with Blobs? Binarypack takes care of everything and will allow you to serialize binary. I'm guessing this is your own thing.

Michelle

On Thu, May 2, 2013 at 10:58 PM, Jesús Leganés Combarro < notifications@github.com> wrote:

Yes, I'm moving JSON strings. El 03/05/2013 06:47, "Michelle Bu" notifications@github.com escribió:

I actually cannot reproduce this. Are you converting your image to a string first?

— Reply to this email directly or view it on GitHub< https://github.com/michellebu/reliable/issues/11#issuecomment-17378465> .

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17379628 .

michelle commented 11 years ago

It's probably good to just build in utf8 as an option for reliable, because runtime really takes a hit in serializing utf8.

piranna commented 11 years ago

Maybe it's my fault :-) I'm using JSON on my project since begin 9 months ago, mainly because it's available everywhere natively and don't need to add any extra library to use it, but I have pending to check other serialization schemes that could be smaller and faster. The fact of using BinaryPack on reliable library it's just and implementation detail for me and should be transparent, since I'm not using it directly. El 03/05/2013 08:00, "Michelle Bu" notifications@github.com escribió:

Why not just directly send JSON objects with Blobs? Binarypack takes care of everything and will allow you to serialize binary. I'm guessing this is your own thing.

Michelle

On Thu, May 2, 2013 at 10:58 PM, Jesús Leganés Combarro < notifications@github.com> wrote:

Yes, I'm moving JSON strings. El 03/05/2013 06:47, "Michelle Bu" notifications@github.com escribió:

I actually cannot reproduce this. Are you converting your image to a string first?

— Reply to this email directly or view it on GitHub< https://github.com/michellebu/reliable/issues/11#issuecomment-17378465> .

— Reply to this email directly or view it on GitHub< https://github.com/michellebu/reliable/issues/11#issuecomment-17379628> .

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17379674 .

piranna commented 11 years ago

Maybe it's the reason why it's so slow for me? El 03/05/2013 08:01, "Michelle Bu" notifications@github.com escribió:

It's probably good to just build in utf8 as an option for reliable, because runtime really takes a hit in serializing utf8.

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17379703 .

piranna commented 11 years ago

http://kentonv.github.io/capnproto/

At last found it :-P I didn't remember the name, but I was thinking about using Cap'n Proto as serialization protocol, or something custom derived from it. Main advantage is that's memory mapped using offsets, so I could transfer the data using arrayBuffer as binaryType and on receiver just use ArrayViewBuffer to access to its content without conversion. What do you think? El 03/05/2013 08:11, "piranna@gmail.com" piranna@gmail.com escribió:

Maybe it's the reason why it's so slow for me? El 03/05/2013 08:01, "Michelle Bu" notifications@github.com escribió:

It's probably good to just build in utf8 as an option for reliable, because runtime really takes a hit in serializing utf8.

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17379703 .

michelle commented 11 years ago

Cap'n Proto was designed with static languages in mind and IMO has little place in Javascript if we're not involving any other lower level languages. The biggest issue I see with it is that it requires a schema--for something like PeerJS it would introduce unnecessary overhead. BinaryPack is actually a modified version of MessagePack, which is pretty widely adopted as a web serialization format.

piranna commented 11 years ago

MessagePack and its improved cousin BinaryPack seems to be better candidates... :-D It only keep to see if there's any implementation using ArrayBuffers :-)

2013/5/3 Michelle Bu notifications@github.com

Cap'n Proto was designed with static languages in mind and IMO has little place in Javascript if we're not involving any other lower level languages. The biggest issue I see with it is that it requires a schema--for something like PeerJS it would introduce unnecessary overhead. BinaryPack is actually a modified version of MessagePack, which is pretty widely adopted as a web serialization format.

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17416094 .

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix." – Linus Tordvals, creador del sistema operativo Linux

michelle commented 11 years ago

Yeah basically there's no need to convert your file blob into a binary string using binary serialization formats, but it's definitely worthwhile to build in the utf8 options, especially since using utf8 on PeerJS with reliable on Chrome will currently not actually serialize utf8.

On Fri, May 3, 2013 at 4:02 PM, Jesús Leganés Combarro < notifications@github.com> wrote:

MessagePack and its improved cousin BinaryPack seems to be better candidates... :-D It only keep to see if there's any implementation using ArrayBuffers :-)

2013/5/3 Michelle Bu notifications@github.com

Cap'n Proto was designed with static languages in mind and IMO has little place in Javascript if we're not involving any other lower level languages. The biggest issue I see with it is that it requires a schema--for something like PeerJS it would introduce unnecessary overhead. BinaryPack is actually a modified version of MessagePack, which is pretty widely adopted as a web serialization format.

— Reply to this email directly or view it on GitHub< https://github.com/michellebu/reliable/issues/11#issuecomment-17416094> .

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix." – Linus Tordvals, creador del sistema operativo Linux

— Reply to this email directly or view it on GitHubhttps://github.com/michellebu/reliable/issues/11#issuecomment-17422823 .