ome / ome2024-ngff-challenge

Project planning and material repository for the 2024 challenge to generate 1 PB of OME-Zarr data
https://pypi.org/project/ome2024-ngff-challenge/
BSD 3-Clause "New" or "Revised" License
11 stars 8 forks source link

Convert labels to v0.5 #29

Closed will-moore closed 3 weeks ago

will-moore commented 1 month ago

Fixes #19

NB: we need to specify shards & chunks for the label conversion (making sure that shards/chunks gives whole numbers) since if we don't then it tries to use the input chunks for labels [1,59,69,136] and the whole array as shard [1,236,275,271] and these are not compatible.

This isn't needed for the image itself, since the chunk shapes for the image are the same as the whole array shape.

Tested on pilot-zarr3-dev with

(bf2raw_env) [wmoore@pilot-zarr3-dev ~]$ time ome2024-ngff-challenge --input-bucket=idr --input-endpoint=https://uk1s3.embassy.ebi.ac.uk --input-anon zarr/v0.4/idr0062A/6001240.zarr 6001240_labels.zarr --output-shards=1,10,512,512 --output-chunks=1,1,256,256

real    1m9.398s
user    0m17.199s
sys 0m3.408s

Uploaded:

$ ./mc cp -r 6001240_labels.zarr uk1s3/idr/share/ome2024-ngff-challenge/0.0.5/

Available at https://deploy-preview-36--ome-ngff-validator.netlify.app/?source=https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240_labels.zarr

Screenshot 2024-08-16 at 16 42 09

joshmoore commented 3 weeks ago

@will-moore: can you look at my few extra commits? This now saves the labels details as well so that they can be edited. e.g.:

diff -C 3 <(jq . < /tmp/6001240.config) /tmp/6001240.config.fixed
*** /dev/fd/63  Tue Aug 20 14:58:11 2024
--- /tmp/6001240.config.fixed   Tue Aug 20 14:55:27 2024
***************
*** 74,82 ****
      ],
      "shards": [
        1,
!       236,
!       275,
!       271
      ]
    },
    "zarr/v0.4/idr0062A/6001240.zarr/labels/0/1": {
--- 74,82 ----
      ],
      "shards": [
        1,
!       59,
!       69,
!       136
      ]
    },
    "zarr/v0.4/idr0062A/6001240.zarr/labels/0/1": {
***************
*** 94,102 ****
      ],
      "shards": [
        1,
!       236,
!       137,
!       135
      ]
    },
    "zarr/v0.4/idr0062A/6001240.zarr/labels/0/2": {
--- 94,102 ----
      ],
      "shards": [
        1,
!       118,
!       69,
!       68
      ]
    },
    "zarr/v0.4/idr0062A/6001240.zarr/labels/0/2": {
will-moore commented 3 weeks ago

Worked fine for me: 👍

$  ome2024-ngff-challenge /Users/wmoore/Desktop/ZARR/data/6001240.zarr 6001240_details.json --output-write-details
$  code 6001240_details.json 
$  ome2024-ngff-challenge /Users/wmoore/Desktop/ZARR/data/6001240.zarr 6001240_from_details.zarr --output-read-details=6001240_details.json
joshmoore commented 3 weeks ago

Great. Moving forward with the next release then.