RFC: Transmission of binary data with JSON

[lilith]

There's already a good writeup on this topic: http://blog.marcinbudny.com/2014/02/sending-binary-data-along-with-rest-api.html

We need to send binary image data, often in excess of 10MB, between the client and server. JSON Base64 increases byte count by 33%, but also has decode/encode overhead.

Key questions for your platform/language:

1. How efficient is your JSON implementation and decoding and encoding enormous base64 strings? With Rust<->Rust we could may see 40ms overhead for 4 codings of a 10MB image.
2. Does your HTTP client support sending multipart requests and parsing multipart responses?

Please speak up if you lack both multipart support and a fast JSON codec.

JSON implementations

... are free to set limits on the maximum size of strings or the message itself.

Vary widely in speed: https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/

Are ubiquitous and usually optimized. Benchmarks of huge string parsing and base64 decoding appreciated.

BSON

BSON has a native binary datatype, but implementations tend to be less optimized than their JSON counterparts. Benchmarks of huge binary blobs appreciated.

HTTP libraries

Benchmarks of multipart parsing overhead on your platform would be appreciated.

When it doesn't matter so much

• We can specify URLs instead of sending input blobs, and offload that fetch (and potentially, have some kind of caching).
• We can ask the server to just send the first image result instead of using multipart. We still have to handle failure JSON, though.

Non-HTTP interactions

• libimageflow users can send pointers to buffers.
• imageflow_tool can read/write to the local filesystem.

[mgaerber]

from my multipart (java) usage I learned the following:

• Java standard libs won't suffice, for good handling you need extra libraries (like Apache http client)
• Do not encode binary content as base64, just send the bytes - the specs are ambiguous when it comes to multipart and so are most implementations
• If you use multipart/form-data you can send requests with multiple files from html forms as well (which is nice for testing and quick integrations)