Resave throws unspecific warning and does nothing

ome / ome2024-ngff-challenge

Project planning and material repository for the 2024 challenge to generate 1 PB of OME-Zarr data

https://pypi.org/project/ome2024-ngff-challenge/

BSD 3-Clause "New" or "Revised" License

11 stars 8 forks source link

Resave throws unspecific warning and does nothing #44

Closed constantinpape closed 2 weeks ago

constantinpape commented 2 weeks ago

I want to convert data from ngff v0.4 to the challenge submission format, but the resave command only creates the output root directory without converting any data. It also throws a warning:

$ ome2024-ngff-challenge resave --cc-by ngff-v2/Platynereis-H2B-TL.ome.zarr ngff-v3/Platynereis-H2B-TL.ome.zarr --output-overwrite

throws

2024-08-30 17:11:09.021 WARNING  no convertible metadata: KeysView(<zarr.attributes.Attributes object at 0x7fe2a1b07350>)

I assume that is because the file is not fully spec compliant with the v0.4 spec (it comes from a custom pipeline), or that the converter makes some additional assumptions not part of the spec.

However, I can't determine what the issue is since the warning thrown here is quite unspecific and since I did not find another way to validate the file yet, see also https://github.com/ome/ome-zarr-py/issues/400.

The file in question is available here: https://drive.google.com/file/d/1WSHQWkOXUfSBahOJdVzrKbj91mczKhZP/view?usp=sharing

joshmoore commented 2 weeks ago

If I run:

ome_zarr view Platynereis-H2B-TL.ome.zarr/c0-t0/

I see the following:

i.e. two separate issues:

the OME-Zarr part starts one level down (no problem really)
it has the wrong dimension separator

will-moore commented 2 weeks ago

The dimension separator is not a problem. However, it did complain if I didn't specify a shard size. This works...

$ ome2024-ngff-challenge resave --cc-by Platynereis-H2B-TL.ome.zarr/c0-t0 zarr_v3/Platynereis-H2B-TL.ome.zarr --output-overwrite --output-shards=512,512,512

Then view it with:

$ ome_zarr view zarr_v3/Platynereis-H2B-TL.ome.zarr

Chunks are loading OK etc:

Screenshot 2024-08-30 at 16 37 20

constantinpape commented 2 weeks ago

Thanks for the replies @joshmoore and @will-moore

Regarding the points you raised:

I can change the dimension separator if needed.
I tried the command with output shards @will-moore , but I still get the same issue as before. What version of the converter tool are you using? I just used the version installed via pip; maybe there are some more recent changes and I should use the version from main instead?

will-moore commented 2 weeks ago

I'm using the released version (or a git branch that is effectively the same). Did you change your command to point at the lower directory /c0-t0 as input? ie. Platynereis-H2B-TL.ome.zarr/c0-t0

constantinpape commented 2 weeks ago

Thanks @will-moore . That did the trick! And I understand the issue now; there was still a conceptual misunderstanding from my side.