views: support for Range requests

[lnielsen]

• Storage.send_file() needs support for specifying a byte range to be sent. Based on the range, it needs to wrap the stream with a proxy which will seek into the file and only send the requested amount of data.
• views.ObjectResource.send_object needs support for parsing Accept-Range

[kbg]

This seems to be related to zenodo/zenodo#1599.

Is there any progress on supporting HTTP/1.1 byte range requests? We are planning to use the upcoming InvenioRDM system, and some of our usecases require the ability to download only parts of large files.

[lnielsen]

There's currently no plans to implement the HTTP range requests in Invenio. We on the other hand plans outsourcing this to the storage system - e.g. if you're using some object store like MinIO with the S3 interface, or is on another cloud enabled storage system. The storage systems are optimized for sending files, and thus we plan on relying on those for the best performance.

[oeway]

Hi, I would like to add some motivation for the range request support here.

We would like to use Zenodo to serve files in Zarr format which has becoming a popular standard for serving massive n-dimensional arrays. On top of zarr, there is also downstream image file format that are built on top of it, e.g. NGFF. Similarily, there is also N5 and cogeo.

Zenodo for example, only allow <100 files in one deposition, HTTP range request support is therefore crucial for serving large files in formats such as Zarr. Essentially, it allows lazy access to large files in open data repository, Python scripts or web applications can immediately benefit from it, and the users won't need to wait for hours to download a large file for a quick glance at the data.

@lnielsen Does this mean it less likely for Zenodo to support HTTP range request in a relatively short term (since I imagine it won't be easy for Zenodo to migrate to other storage backend)?

[bgruening]

I think this is now working. The only missing part is that Invenio is not exposing the accept-ranges: bytes header.

[joshmoore]

Thanks for the update, @bgruening!