search

seung-lab / igneous

Scalable Neuroglancer compatible Downsampling, Meshing, Skeletonizing, Contrast Normalization, Transfers and more.
GNU General Public License v3.0
40 stars 16 forks source link

uncompressed sharded format #152

Closed tannerBMC closed 1 year ago

tannerBMC commented 1 year ago

I'm using igneous version 4.15.2 and create a sharded volume from a non-sharded volume via create_image_shard_transfer_tasks. It works and I can view it via neuroglancer. Thanks a lot!

Howerver I'm unable to view it with the siibra-explorer in the current form. One issue is that they use dataproxy, which does not support compressed data. Using encoding='raw', and the undocumented parameter compress=None, I managed to get "data_encoding": "raw",

However I cannot modify "minishard_index_encoding": "gzip", as this is not supported, i.e. _encodingTypeError: create_image_shard_transfer_tasks() got an unexpected keyword argument 'minishard_index_encoding'

Is there a way to generate sharded files where everything is uncompressed?

william-silversmith commented 1 year ago

Hi @tannerBMC, I pushed an update in version 4.17.0. Would you mind giving that a try? You can now set minishard_index_encoding="raw".

tannerBMC commented 1 year ago

Hi William

Thanks a lot for your fast response and adaption. Most appreciated.

Looks good.

I tested it and can create in this way info files which state "minishard_index_encoding": "raw" and the chunk files increased in size by 17 bits. I can load the generated chunks into neuroglancer without problems.

Unfortunately it did not solve my problem with loading the sharded chunks into siibra-explorer (https://atlases.ebrains.eu/viewer/). They might be use an earlier version of sharded chunk file format.

Best regards, Christine Tanner

william-silversmith commented 1 year ago

Thanks for the kind words. If you get in touch with the siibra people, you can let them know that I'd be willing to give them some advice on supporting the sharded format. Sorry you're running into trouble!

Will

On Mon, Jun 5, 2023, 12:27 PM Christine Tanner @.***> wrote:

Hi William

Thanks a lot for your fast response and adaption. Most appreciated.

Looks good.

I tested it and can create in this way info files which state "minishard_index_encoding": "raw" and the chunk files increased in size by 17 bits. I can load the generated chunks into neuroglancer without problems.

Unfortunately it did not solve my problem with loading the sharded chunks into siibra-explorer (https://atlases.ebrains.eu/viewer/). They might be use an earlier version of sharded chunk file format.

Best regards, Christine Tanner

— Reply to this email directly, view it on GitHub https://github.com/seung-lab/igneous/issues/152#issuecomment-1577110984, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATGQSKCN7ZP5C44T6LT2OLXJYCIHANCNFSM6AAAAAAY2VWMPY . You are receiving this because you commented.Message ID: @.***>