search

AllenInstitute / biofile-finder

An open-use web application created for easy access, collaboration, and sharing of datasets through rich metadata search, filter, sort, and direct viewing in common industry applications or in our web-based 3D Volume Viewer.
https://biofile-finder.allencell.org/
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

feature/web-download-threshold #314

Closed BrianWhitneyAI closed 1 week ago

BrianWhitneyAI commented 3 weeks ago

The purpose of this PR is to resolve #283. In web browsers we have to read the entire zarr image into memory and cannot stream files as zarrs are a full directory. As a result of this we are bound to the browsers inherent memory limit of about 2GB. We should notify users if they are trying to download a file larger than this threshold.

BrianWhitneyAI commented 3 weeks ago

Screenshot 2024-11-01 at 4 08 03 PM

SeanLeRoy commented 3 weeks ago

Also, we should use the "File Size" metadata field when we can (available on the file object as the "size" property) and only calculate the file size when that isn't provided

BrianWhitneyAI commented 2 weeks ago

I disabled the download button on web for zarrs larger than 2 GB and had the size default to fileDetail.Size. Notably we internally set size to 0 if size is not present.

BrianWhitneyAI commented 2 weeks ago

I left intact the download error message since users could batch select files and download them using the download option, bypassing the file detail download button.

SeanLeRoy commented 2 weeks ago

I left intact the download error message since users could batch select files and download them using the download option, bypassing the file detail download button.

Can we disable that ahead of time as well?

BrianWhitneyAI commented 2 weeks ago

I left intact the download error message since users could batch select files and download them using the download option, bypassing the file detail download button.

Can we disable that ahead of time as well?

What would that look like? just if any large zarr is selected it cannot be downloaded? I think this might be confusing to users who select a bunch of files but are blocked by a single zarr file.

SeanLeRoy commented 2 weeks ago

I left intact the download error message since users could batch select files and download them using the download option, bypassing the file detail download button.

Can we disable that ahead of time as well?

What would that look like? just if any large zarr is selected it cannot be downloaded? I think this might be confusing to users who select a bunch of files but are blocked by a single zarr file.

I think just the focused file is downloaded the moment so you'd just need to calculate that one which you'd already be doing / have done for the filedetails pane