search

IDR / idr-metadata

Curated metadata for all studies published in the Image Data Resource
https://idr.openmicroscopy.org
14 stars 24 forks source link

idr0001-graml-sysgro to NGFF #683

Open will-moore opened 6 months ago

will-moore commented 6 months ago

idr0001 has 192 x 96-Well Plates, 6 acquisitions each. 1 Plate converted below is 47 GB. Approx 4.5 TB in total. bioformats2raw took ~30mins to convert 1 Plate => approx 4 days in total.

NB: The need to convert multi-acquisition Flex data (idr0001) is because the support for that hasn't been ported from IDR to mainline BioFormats: https://github.com/ome/bioformats/pull/3537

will-moore commented 6 months ago

convert first plate for testing... on pilot-zarr1-dev

~/bioformats2raw-0.6.0-24/bin/bioformats2raw "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/Meas_01(2012-07-31_10-41-12)/001001001.flex" JL_120731_S6A
...
2024-01-08 13:36:06,658 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,661 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,664 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,668 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,671 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
...

(at 2pm)...

Done about 14:15 (approx 45 mins).

will-moore commented 6 months ago

make bucket etc... then:

(base) [wmoore@pilot-zarr1-dev ~]$ ./mc cp -r /data/idr0001/JL_120731_S6A.ome.zarr uk1s3/idr0001/zarr
...0731_S6A.ome.zarr/OME/METADATA.ome.xml: 96.57 GiB / 96.57 GiB ━━━━━━━━━━━━━━ 48.81 MiB/s 33m45s
will-moore commented 6 months ago

Looks like we are getting duplicates of each of the 6 acquisitions for each Plate. We would expect each Well to have 6 Fields but we get 12 (6 pairs of duplicates): https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/

Image

sbesson commented 6 months ago

There are a series of issues we might need to review here. For reference, the source data corresponding to the example above is https://ftp.ebi.ac.uk/pub/databases/IDR/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/.

As can be seen above, there are 6 measurement folders, which are interpreted as plate acquisitions using the IDR Flex reader. Looking at the file listing for each measurement, there are 2 fields of views per well and plate acquisition.

001001001.flex
001001002.flex
001002001.flex
001002002.flex
...

In IDR, each well of each plate acquisition only contains 1 field per view and the fileset only includes the Flex files ending with 001.flex. I cannot comment know whether this is to be treated as an issue or an active decision made during the loading of this historical study /cc @joshmoore

If the intent is to create a Zarr dataset matching the current IDR representation, I think of two options:

will-moore commented 5 months ago

Let's try omero-cli-zarr export for comparison... On idr-ftp... Update to latest https://github.com/ome/omero-cli-zarr/pull/147

conda activate omero_zarr_export
pip uninstall omero-cli-zarr

# cloned https://github.com/ome/omero-cli-zarr/pull/147 and merged https://github.com/ome/omero-cli-zarr/pull/156
$ cd /data/idr0001/omero_cli_zarr/omero-cli-zarr
$ pip install -e .

$ omero zarr export Plate:2552 --name_by name

...export status... 5 out of 96 Wells done in 35 mins. 7 mins per Well is 11 hours per Plate!

... completed at 22:08 - ~11.5 hours.

Upload...

./mc cp -r /data/idr0001/omero_cli_zarr/JL_120731_S6B.ome.zarr uk1s3/idr0001/zarr

Looks good at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr/

will-moore commented 5 months ago

Tried exporting Polygons as labels... but this fails as the Polygons are overlapping:

(omero_zarr_export) [wmoore@pilot-zarr1-dev omero_cli_zarr]$ omero zarr polygons Plate:2552 --source-image=JL_120731_S6B.ome.zarr

Found 51 mask shapes in 51 ROIs
Unique dimensions: {'T': {None}, 'C': {None}, 'Z': {None}}
source_image JL_120731_S6B.ome.zarr/D/7/0
Ignoring dimensions {'C', 'T', 'Z'}
Traceback (most recent call last):
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 125, in _wrapper
    return func(self, *args, **kwargs)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 333, in polygons
    plate_shapes_to_zarr(plate, ["Polygon"], args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 113, in plate_shapes_to_zarr
    saver.save(list(masks.values()), args.label_name)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 334, in save
    labels, fill_colors, properties = self.masks_to_labels(
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 565, in masks_to_labels
    raise Exception(
Exception: Shape 624314 overlaps with existing labels
will-moore commented 5 months ago

Run mkngff on idr0125-pilot with 

$ sudo mkdir /idr0001 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0001 /idr0001
$ ls /idr0001/zarr/
JL_120731_S6A.ome.zarr  JL_120731_S6B.ome.zarr

as omero-server

omero mkngff sql 16452 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr" "/idr0001/zarr/JL_120731_S6B.ome.zarr" > "idr0001/16452.sql"
$ psql -U omero -d idr -h $DBHOST -f idr0001/16452.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        5289241
(1 row)

COMMIT

$ omero mkngff symlink /data/OMERO/ManagedRepository 16452 "/idr0001/zarr/JL_120731_S6B.ome.zarr" --bfoptions
...
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr -> /idr0001/zarr/JL_120731_S6B.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr.bfoptions

http://localhost:1040/webclient/?show=image-1230224 ...

Fails with:

2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Unregistered servant:078c4a67-66a5-46f8-822f-8e196ea78f82/a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery(omero.api._IQueryTie@cbaf0b18)
2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Removed servant from adapter: a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1704877049382] time[216572] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1704876989274] time[276679] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-10) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,957 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
java.lang.RuntimeException: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
will-moore commented 5 months ago

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79 but that's not where the ClassCastException comes from.

sbesson commented 5 months ago

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79 but that's not where the ClassCastException comes from.

I suspect the error comes from the underlying reader and is rethrown from the Bio-Formats pixel buffer. Is there more information in the following lines of the stack trace?

will-moore commented 5 months ago

Ah, yes...

Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:755)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:353)
        at loci.formats.FormatReader.setId(FormatReader.java:1443)
        at loci.formats.ImageReader.setId(ImageReader.java:849)
        at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.Memoizer.setId(Memoizer.java:690)
        at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
        ... 82 common frames omitted
will-moore commented 5 months ago

That line is https://github.com/ome/ZarrReader/blob/main/src/loci/formats/in/ZarrReader.java#L755

String acqStartTime = (String) acquistion.get("starttime");

The schema states that this is an integer: https://github.com/ome/ngff/blob/main/0.4/schemas/plate.schema#L35

cc @dgault

will-moore commented 5 months ago

We want to create a copy of the original data without ...2.flex files but without duplicating all the data!

On pilot-zarr1-dev

export PLATE=JL_120801_S7A
cd /data/idr0001/raw

mkdir $PLATE
cd $PLATE
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
  mkdir "$i"
  for f in $(ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i"); do
    ln -s "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i/$f" "$i/$f"
  done
  rm "$i"/*2.flex
done

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw "JL_120801_S7A/Meas_01(2012-08-01_10-57-32)/001001001.flex"  ngff/JL_120801_S7A.ome.zarr
...

Conversion took approx 30 mins.

(bioformats2raw) [wmoore@pilot-zarr1-dev ngff]$ ~/mc cp -r /data/idr0001/raw/ngff/JL_120801_S7A.ome.zarr uk1s3/idr0001/zarr
...7A.ome.zarr/OME/METADATA.ome.xml: 47.15 GiB / 47.15 GiB ━━━━━━━━━━━━━━━━ 49.82 MiB/s 16m9s
will-moore commented 5 months ago 
omero mkngff sql 16454 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120801_S7A.ome.zarr" "/idr0001/zarr/JL_120801_S7A.ome.zarr" > "idr0001/16454.sql"
$ psql -U omero -d idr -h $DBHOST -f idr0001/16454.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        5289242
(1 row)

COMMIT

$ omero mkngff symlink /data/OMERO/ManagedRepository 16454 "/idr0001/zarr/JL_120801_S7A.ome.zarr" --bfoptions
...
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff/JL_120801_S7A.ome.zarr -> /idr0001/zarr/JL_120801_S7A.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff/JL_120801_S7A.ome.zarr.bfoptions

This Plate renders OK in webclient 👍 NB: it avoids the ZarrReader bug above since it doesn't have acquisition starttime or endtime metadata.

will-moore commented 5 months ago

Create symlinks...

cd /data/idr0001/raw

for PLATE in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/); do
echo $PLATE
mkdir $PLATE
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
  mkdir "$PLATE/$i"
  for f in $(ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i"); do
    ln -s "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i/$f" "$PLATE/$i/$f"
  done
  rm -f "$PLATE/$i"/*2.flex
done
done

Count flex files for each Plate/Acquisition in plates.csv (in IDR):

for PLATE in $(cat plates.csv); do
echo $PLATE >> counts3.log
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
 ls "$PLATE/$i" | wc >> counts3.log
done
done

cat counts3.log grep -v 96
# ignored Plates where all acquisitions have 96 files. That leaves...
JL_120801_S7B
     95      95    1425
JL_120814_S17A
     95      95    1425
JL_120816_S19B
     95      95    1425
JL_120907_S2A
     95      95    1425
JL_121214_J2_1
     95      95    1425
JL_121217_J7_1
     95      95    1425
JL_121219_J4_3
     95      95    1425
JL_121220_J5_3
     95      95    1425
JL_130126_J10_3
     93      93    1395
JL_130126_J10_4
     91      91    1365
JL_130126_J9_3
     94      94    1410
JL_130126_J9_4
     93      93    1395
JL_130127_J10_5
     94      94    1410
JL_130127_J9_5
     93      93    1395
JL_130127_J9_6
     94      94    1410
JL_130128_J9_7
     95      95    1425
JL_130128_J9_8
     95      95    1425
     95      95    1425
     95      95    1425
     95      95    1425
     95      95    1425
JL_130304_R1_3
     95      95    1425
X_110213_S2_Blue1000
     95      95    1425
X_110331_S13
     95      95    1425
X_110425_S6
     95      95    1425
X_110429_S24
     95      95    1425
will-moore commented 5 months ago

Checking for counts of ...2.flex files which I've "deleted" (removed symlinks) above. Most plates/acquisitions have 0 or 96 but some have other counts. Getting complex!

for PLATE in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/); do
echo $PLATE >> counts2.log
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
 ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i" | grep "2.flex" | wc >> counts2.log
done
done
will-moore commented 5 months ago

On pilot-zarr1-dev


$ cd /data/idr0001/raw
$ cat plates.csv
JL_120731_S6A
JL_120731_S6B
JL_120801_S7A
...
$ cat plates.csv | wc
    192     192    2828

for i in $(cat plates.csv); do
  first_flex=$(find $i -name *.flex -print -quit)
  echo $first_flex
  ~/bioformats2raw-0.6.0-24/bin/bioformats2raw "$first_flex"  ngff/$i.ome.zarr
done
will-moore commented 5 months ago

Try viewing some of the 22 Plates above which have fewer than 96 .flex files... Are these OK on IDR??

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/19-56-57.181/Meas_01(2013-01-26_16-42-53)/005004001.flex
}
will-moore commented 5 months ago

Since none of the Plates with missing .flex files above have successfully been converted by the script running above, let's test one on it's own (NB: this is the symlinked data with the *2.flex files removed):

screen -S idr0001_test:

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw JL_120801_S7B/Meas_01\(2012-08-01_18-40-47\)/001001001.flex ngff/JL_120801_S7B.ome.zarr
...
2024-01-12 13:51:27,012 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008010001.flex
2024-01-12 13:51:27,669 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008011001.flex
2024-01-12 13:51:28,343 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008012001.flex
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@16150369): java.lang.ArithmeticException: / by zero
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2192)
Caused by: java.lang.ArithmeticException: / by zero
        at loci.formats.in.FlexReader.groupFiles(FlexReader.java:1402)
        at loci.formats.in.FlexReader.initFlexFile(FlexReader.java:581)
        at loci.formats.in.FlexReader.initFile(FlexReader.java:390)
        at loci.formats.FormatReader.setId(FormatReader.java:1389)
        at loci.formats.ImageReader.setId(ImageReader.java:849)
        at com.glencoesoftware.bioformats2raw.Converter.getBaseReaderClass(Converter.java:2032)
        at com.glencoesoftware.bioformats2raw.Converter.convert(Converter.java:540)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:516)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:107)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        ... 9 more

Also try it on the original data where the number of 2.flex files for each acquisition matches the *1.flex file counts.

(bioformats2raw) [wmoore@pilot-zarr1-dev raw]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120801_S7B/Meas_01\(2012-08-01_18-40-47\)/001001001.flex ngff/JL_120801_S7B.ome.zarr

This actually worked and has the correct number of Fields!

(base) [wmoore@pilot-zarr1-dev ngff]$ ls -alh !$
ls -alh JL_120801_S7B.ome.zarr/A/1
total 0
drwxrwxr-x.  8 wmoore wmoore  88 Jan 12 14:28 .
drwxrwxr-x. 14 wmoore wmoore 169 Jan 12 14:13 ..
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:12 0
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:15 1
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:18 2
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:21 3
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:24 4
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:28 5

Plate has the same number of images (.zattrs) as previous S7A plate!

$ find JL_120801_S7A.ome.zarr/ -name ".zattrs" | wc
    674     674   24906
$ find JL_120801_S7B.ome.zarr/ -name ".zattrs" | wc
    674     674   24906
will-moore commented 5 months ago

Another Plate that failed with symlinked data (without *2.flex files) - run against original data...

(bioformats2raw) [wmoore@pilot-zarr1-dev raw]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120814_S17A/Meas_01\(2012-08-14_10-49-32\)/001001001.flex ngff/JL_120814_S17A.ome.zarr
...
will-moore commented 1 week ago

5 months later

Investigation of possible Bio-Formats fix at https://github.com/ome/bioformats/pull/3537 finds that this won't be easy/viable solution. So we need to revive the NGFF conversion work...

Summary from first read of history above:

will-moore commented 3 days ago

If we need to use omero-cli-zarr, old idr is still available at ssh 45.88.81.175 etc...