Closed dominictarr closed 9 years ago
Okay so the problem JSON is that it sucks for binary. If we wanted to send a gzipped tarball or a png we'd have to base64 encode it, and that feels dumb... (although, meatspace works with base64 images and is cool)
I was thinking we'd have to use a binary framing around a text encoding... but substack's readme shows a more brutally simple approach. if you use the hash of the object as the delimiters for that object, that is the one string that cannot be inside that object... because the hash did not exist before the object was created. this means you could have a text protocol that supports binary without needing length delimiters... so for plain text objects, it will be readable.
Turns out this isn't actually how substack's module works, but this would work.
hmm. Are length delimeters even that bad? I guess we could also use the redis protocol that supports binary content, with length delimiters and is human readable if the objects are just text.
implemented.
issues to investigate
this is a nice method that works around the whitespace problem: https://camlistore.googlesource.com/camlistore/+/master/doc/json-signing/json-signing.txt
maybe we should do that, and keep the encoded form around so that "reencoding" is just returning that string.
Also, if all objects can be a single line, then we can just use line separated json, which is much cheaper than a streaming json parser which must be implemented in js.