search

AllenInstitute / biofile-finder

An open-use web application created for easy access, collaboration, and sharing of datasets through rich metadata search, filter, sort, and direct viewing in common industry applications or in our web-based 3D Volume Viewer.
https://biofile-finder.allencell.org/
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

Add threshold for web download (2 GB) warning message #283

Closed BrianWhitneyAI closed 1 week ago

BrianWhitneyAI commented 1 month ago

In web browsers we have to read the entire zarr image into memory and cannot stream files as zarrs are a full directory. As a result of this we are bound to the browsers inherent memory limit of about 2GB. We should notify users if they are trying to download a file larger than this threshold.

toloudis commented 1 month ago

Just curious, where do you come up with the 2GB limit value? Is this a known thing?

BrianWhitneyAI commented 1 month ago

I havent seen a hard threshold. It looks like each browser is a little different but have similar limitations on the ram that they can use. I found failures for Safari, Chrome and Firefox between 1.8 and 2.4 GB download progress. (We may want to just say 1GB then). Ideally we find a better way to stream download zarrs in web browsers as right now we have to read the full file into memory.

I did get this responose from the browsers AI

""" Chrome: While Chrome has no hard limit for how much memory it can use, it is bound by the system's available memory. Chrome can run into issues with files over 2GB due to memory allocation limits in JavaScript for certain types of objects, even on systems with more available RAM. Firefox: Like Chrome, Firefox also doesn’t impose a specific limit, but practical limits come from system memory and performance degradation. There are reports of issues with files over 2GB. Safari: Safari has better memory management on macOS, but it shares similar limitations to other browsers in terms of downloading large files. It's optimized for macOS’s memory handling, but if too much memory is consumed, it can cause the browser to crash or the system to slow down. Edge: Edge behaves similarly to Chrome since both are based on the Chromium engine, but there may be some minor differences in memory handling. """ and a10639e6-c69e-4a4f-9b2a-3c945df466c4