zenodo / zenodo

Research. Shared.
https://zenodo.org
GNU General Public License v2.0
917 stars 243 forks source link

Loosen rate limit for chunk requests #2247

Open oeway opened 2 years ago

oeway commented 2 years ago

Is your feature request related to a problem? Please describe.

With the recent support of range requests, we can use web apps to visualize large data files directly from zenodo, e.g. using zarr file format. However, due to very restricted rate limit set by zenodo, the viewer fails easily when sending many requests for small chunks and the server returns the following error:

Status Code: 429 TOO MANY REQUESTS

Describe the solution you'd like

Loosen the rate limit for range requests.

Additional context

To reproduce the issue, you can try it with this viewer example: https://imjoy.io/lite?plugin=https://gist.githubusercontent.com/oeway/c0d4ff439bb738a300da34ec7e1f77b9/raw/VizarrReferenceStore-Zenodo.imjoy.html

If successful, you should see an image open, but when you change the slider position to view other plane of the image stack, it fails and you can see the "TO MANY REQUESTS" error in the developer tool.

cc @rabernat

oeway commented 2 years ago

FYI: this is another example where we try to use zenodo to host multi-resolution images but failed because of this rate limit: https://imodpasteur.github.io/shareloc-utils/shareloc-potree-viewer.html?load=https://sandbox.zenodo.org/api/files/46c189b4-02a7-4eb6-b677-62750daeb971/7312e0.zip

(The image will open, but only some initial chunks, and failed afterwards)

joshmoore commented 1 year ago

If of interest, please see https://biorxiv.org/content/10.1101/2023.02.17.528834v4 for background on the use of the format that @oeway describes (Zarr) for use in the bioimaging domain. One of the primary benefits of the format is reducing the need to download an entire dataset. E.g., this image is 500 GB for what would otherwise by a single TIFF file, but as you can see the browser can access only the necessary portions of the data to provide quicker visualization. The trade-off for this reduction in bandwidth, of course, is that many more requests are made to the server.