search

NOAA-OWP / DMOD

Distributed Model on Demand infrastructure for OWP's Model as a Service
Other
7 stars 15 forks source link

Ensure correct handling of binary data upload and download for datasets #539

Open robertbartel opened 8 months ago

robertbartel commented 8 months ago

Issues #338 and #538 are likely to have highly interrelated solutions.

Edit: removed self-referential mention of #539.

aaraney commented 6 months ago

To add a little context to this issue, the current capabilities of DMOD enable uploading and downloading data to and from the data service over a websocket connection using JSON as the messaging format. JSON does not support encoding and decoding binary data. This causes problems in the context of uploading binary data to the data service. websocket's do support transferring arbitrary binary data, however given how we choose to encode and transfer data in the current scheme, we are unable to transfer arbitrary binary data.

There are a slew of avenues we can pursue to remedy this problem (e.g. base64 or base85 encoding, custom binary format). However, at this point, I wanted to just state the problem.

robertbartel commented 6 months ago

Blocked by #575.