Handling large binary data transfers

[jayrbolton]

For example, you might have an HTTP endpoint for uploading images. Typically, this would be handled with a multipart/form-data request. What would be the best way to handle this with an OpenRPC API?

[github-actions[bot]]

Welcome to OpenRPC! Thank you for taking the time to create an issue. Please review the guidelines

[BelfordZ]

Hi @jayrbolton

Thank you for the question. JSON-RPC makes no suggestions over this type of http-specific challenge.

That being said, perhaps you could construct a custom http transport that handles batch requests as multipart, and then have an open-rpc method called something like upload-big-file-chunk ?

I'm not exactly sure what the best way to handle this would be since json-rpc and thus open-rpc is meant to be transport agnostic.

please feel free to share any solutions you come across for dealing with this. Thanks!

[jayrbolton]

Hey @BelfordZ, I had similar thoughts. Here are my ideas in another issue: https://github.com/kbaseIncubator/jsonrpcbase/issues/20

In the issue, I mention leaning towards the CBOR/BSON way, because it keeps the transport agnosticism. But I'm still not crazy about it because I think it'd be pretty tricky to stream only one nested param to a file without loading the whole payload into memory.

I feel like the multi-part way would be easier to stream the binary data to a file, without the surrounding json structure, and that seems important. But it would be HTTP-specific.

I do realize this is not the responsibility of the Open-RPC layer, but is more of a tangential problem. Feel free to close if you'd like.

[shanejonas]

Just some more info to add to the thread:

I'd recommend base64 as its used in web more often, not sure how they compare.

you can validate non-json data like base64 for example, or probably your own, with JSON-Schema:

https://json-schema.org/understanding-json-schema/reference/non_json_data.html

[jayrbolton]

Nice, I never saw that part of json-schema, that feature is definitely a plus here.

A couple downsides of embedding base64: you're increasing the size of your file data and the payload, which definitely matters for large uploads; it's still tricky to stream only the upload content to a local file on the server without loading any surrounding rpc structure into memory.

[BelfordZ]

@jayrbolton All the good solutions I can think of are a bit departed from JSON-RPC 2.0. Perhaps its something to add to the list of things to support in JSON-RPC 3.0 whenever we gather the courage to put that together =P

Any new ideas about this since above?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.