google / neuroglancer

WebGL-based viewer for volumetric data
Apache License 2.0
1.07k stars 295 forks source link

Fallback when reading sharded data from a host that does not support range requests #619

Open d-v-b opened 2 months ago

d-v-b commented 2 months ago

Apropos #606, not every host supports byte range requests to the same degree (or at all). When fetching chunks from a sharded datasource, if byte range requests fail, Neuroglancer displays no data at all. But in this case Neuroglancer could fall back to fetching the entire shard file and reading chunks in memory. This will fail for large shards, but with the same familiar failure mode triggered by any object that's too large to fetch. For small shards, however, fetching a full shard as a fallback would allow neuroglancer to display data that it otherwise could not.

jbms commented 2 months ago

I can imagine that in some cases this could be useful. But on the other hand, it isn't clear how often it would really be useful --- I imagine in many cases having to download the entire shard would lead to a very poor experience if it fits in memory at all. Certainly if this were supported, it would be important to surface a warning to the user somehow, to indicate that it is operating in a degraded mode and that they should fix the web server.

A simpler improvement would be to surface a more informative error when range requests aren't supported but e.g. a HEAD request succeeds. Related to this, it would be useful to surface the chunk download errors outside of the javascript console...

d-v-b commented 2 months ago

Yes, the net utility of falling back to fetching the whole shard depends on the empirical question of what the typical shard size is. If nearly everyone deploys 10 GB shards, then it's not practical to consider fetching that. But 100 MB shards, on a local connection, would be technically feasible, albeit not a great user experience.