Open oeway opened 2 years ago
FYI: this is another example where we try to use zenodo to host multi-resolution images but failed because of this rate limit: https://imodpasteur.github.io/shareloc-utils/shareloc-potree-viewer.html?load=https://sandbox.zenodo.org/api/files/46c189b4-02a7-4eb6-b677-62750daeb971/7312e0.zip
(The image will open, but only some initial chunks, and failed afterwards)
If of interest, please see https://biorxiv.org/content/10.1101/2023.02.17.528834v4 for background on the use of the format that @oeway describes (Zarr) for use in the bioimaging domain. One of the primary benefits of the format is reducing the need to download an entire dataset. E.g., this image is 500 GB for what would otherwise by a single TIFF file, but as you can see the browser can access only the necessary portions of the data to provide quicker visualization. The trade-off for this reduction in bandwidth, of course, is that many more requests are made to the server.
Is your feature request related to a problem? Please describe.
With the recent support of range requests, we can use web apps to visualize large data files directly from zenodo, e.g. using zarr file format. However, due to very restricted rate limit set by zenodo, the viewer fails easily when sending many requests for small chunks and the server returns the following error:
Describe the solution you'd like
Loosen the rate limit for range requests.
Additional context
To reproduce the issue, you can try it with this viewer example: https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/c0d4ff439bb738a300da34ec7e1f77b9/raw/VizarrReferenceStore-Zenodo.imjoy.html
If successful, you should see an image open, but when you change the slider position to view other plane of the image stack, it fails and you can see the "TO MANY REQUESTS" error in the developer tool.
cc @rabernat