search

scalableminds / webknossos

Visualize, share and annotate your large 3D images online
https://webknossos.org
GNU Affero General Public License v3.0
126 stars 23 forks source link

Zarr Streaming - hand through compressed buckets #6180

Open leowe opened 2 years ago

leowe commented 2 years ago

Detailed Description

Follow-up for #6144.

When using zarr remote datasets, it is possible to define a compression scheme in the .zarray file.

Currently, this feature is unused and all requested buckets are decompressed before sending them to the client. We should hand through the compressed files if possible saving time de-compressing and on the transmission. For that it is necessary to write new data request code paths that don't decompress the read buckets.

normanrz commented 2 years ago

This would be a great addition. In a first iteration, I would only apply this optimization to datasets, where the stored chunk size matches the output chunk size. Otherwise we could create too much server load.