search

glencoesoftware / raw2ometiff

Raw format to OME-TIFF converter
GNU General Public License v2.0
49 stars 22 forks source link

broken pipeline bioformats2raw and raw2ometiff in 0.3.0-rc3? #62

Closed BioinfoTongLI closed 3 years ago

BioinfoTongLI commented 3 years ago

Hello,

I am trying to rebuild the pyramid that is already an ome.tif by converting all through the bioformats2raw and raw2ometiff pipeline.

And I am getting this error:

  Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.pyramid.PyramidFromDirectoryWriter@8646db9): java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at picocli.CommandLine.executeUserObject(CommandLine.java:1792)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2349)
    at picocli.CommandLine.parseWithHandler(CommandLine.java:2284)
  Causedat picocli.CommandLine.call(CommandLine.java:2560)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.main(PyramidFromDirectoryWriter.java:210)
  Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.initialize(PyramidFromDirectoryWriter.java:726)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:225)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:97)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
  rk dir... 9 more

Command line used for bf2raw: /bf2raw/bioformats2raw-0.3.0-rc3/bin/bioformats2raw --max_workers 20 --resolutions 5 --no-hcs $ome_tif "${stem}" and raw2ometiff: /raw2tif/raw2ometiff-0.3.0-rc3/bin/raw2ometiff --max_workers 20 "${stem}" [new.tif]

PS: the output of bf2raw is a folder with 0 and OME which seems good to me.

Any idea what I did wrong?

Many thanks for your help!

Best, Tong

melissalinkert commented 3 years ago

Thanks for reporting this, @BioinfoTongLI. Do you see a similar error if the --no-hcs option is removed from the bioformats2raw step? Were there any error or warning messages in the output of bioformats2raw?

BioinfoTongLI commented 3 years ago

Sorry I should have mentioned that - yeah. I tried without. Same error message.

  Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.pyramid.PyramidFromDirectoryWriter@8646db9): java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at picocli.CommandLine.executeUserObject(CommandLine.java:1792)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2349)
    at picocli.CommandLine.parseWithHandler(CommandLine.java:2284)
  Causedat picocli.CommandLine.call(CommandLine.java:2560)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.main(PyramidFromDirectoryWriter.java:210)
  Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.initialize(PyramidFromDirectoryWriter.java:726)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:225)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:97)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
  rk dir... 9 more

and yes. There was a warning for bioformats2raw

OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp6105402081124591895/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix t
he stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
BioinfoTongLI commented 3 years ago

additional information: output of tiffcomment 

<?xml version="1.0" encoding="UTF-8" ?><!-- Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: https://docs.openmicroscopy.org/latest/ome-model/ome-tiff/. --><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd" UUID="urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832"><Plate ID="Plate:0" Name="RV_GON_01A5" ColumnNamingConvention="letter" RowNamingConvention="number" Columns="4" Rows="1" ExternalIdentifier="6f8a58d4-5b1e-4ecc-93fd-824a8d2c3468"><Description>PerkinElmer Slide Holder H.45 (up to 4 Slides), Sample below Slide</Description><Well ID="Well:0" Row="1" Column="1"><WellSample ID="WellSample:0:1" Index="1" PositionX="900.911" PositionXUnit="µm" PositionY="-15916.0885" PositionYUnit="µm"><ImageRef ID="Image:0" /></WellSample></Well></Plate><Instrument ID="Instrument:0"><Microscope Manufacturer="PerkinElmer" Model="Phenix" /><Detector ID="Detector:0:1" Model="AndorZylaCam" /><Detector ID="Detector:0:2" Model="AndorZylaCam" /><Detector ID="Detector:0:3" Model="AndorZylaCam" /><Detector ID="Detector:0:4" Model="AndorZylaCam" /><Detector ID="Detector:0:5" Model="AndorZylaCam" /><Objective ID="Objective:0" NominalMagnification="40" LensNA="1.1" WorkingDistanceUnit="mm" /></Instrument><Image ID="Image:0"><AcquisitionDate>2021-04-28T11:00:11.1544884+01:00</AcquisitionDate><InstrumentRef ID="Instrument:0" /><ObjectiveSettings ID="Objective:0" /><Pixels ID="Pixels:0" DimensionOrder="XYZCT" Type="uint16" BigEndian="false" Interleaved="false" SizeX="35033" SizeY="30924" SizeC="5" SizeZ="1" SizeT="1" PhysicalSizeX="0.146165685373" PhysicalSizeY="0.146165685373"><Channel ID="Channel:0:1" Color="6750207" Name="DAPI" SamplesPerPixel="1" EmissionWavelength="456" ExcitationWavelength="375" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:1" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:2" Color="16711935" Name="Opal 520" SamplesPerPixel="1" EmissionWavelength="522" ExcitationWavelength="488" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:2" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:3" Color="12648703" Name="Opal 570" SamplesPerPixel="1" EmissionWavelength="599" ExcitationWavelength="561" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:3" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:4" Color="4539903" Name="Opal 650" SamplesPerPixel="1" EmissionWavelength="706" ExcitationWavelength="640" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:4" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:5" Color="14614527" Name="Atto 425" SamplesPerPixel="1" EmissionWavelength="482" ExcitationWavelength="425" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:5" Binning="1x1" /><LightPath /></Channel><TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="1" FirstT="0" FirstZ="0" IFD="1" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="2" FirstT="0" FirstZ="0" IFD="2" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="3" FirstT="0" FirstZ="0" IFD="3" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="4" FirstT="0" FirstZ="0" IFD="4" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><Plane ExposureTime="0.08" TheC="0" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="1" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="2" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="3" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="4" TheZ="0" TheT="0" /></Pixels></Image><StructuredAnnotations><MapAnnotation ID="Annotation:Resolution:0" Namespace="openmicroscopy.org/PyramidResolution"><Value><M K="1">17516 15462</M><M K="2">8758 7731</M><M K="3">4379 3865</M><M K="4">2189 1932</M><M K="5">1094 966</M><M K="6">547 483</M></Value></MapAnnotation></StructuredAnnotations></OME>

and output of tiffinfo

TIFF Directory at offset 0x1101b6c9c (4565200028)
  Image Width: 35033 Image Length: 30924
  Tile Width: 512 Tile Length: 512
  Resolution: 68417.3, 68417.3 pixels/cm
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: Deflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 7548
  Planar Configuration: single image plane
  SubIFD Offsets: 1001138844 1051726740 1064769818 1068757854 1069588490 1069800460
  ImageDescription: <?xml version="1.0" encoding="UTF-8" ?><!-- Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: https://docs.openmicroscopy.org/latest/ome-model/ome-tiff/. --><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd" UUID="urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832"><Plate ID="Plate:0" Name="RV_GON_01A5" ColumnNamingConvention="letter" RowNamingConvention="number" Columns="4" Rows="1" ExternalIdentifier="6f8a58d4-5b1e-4ecc-93fd-824a8d2c3468"><Description>PerkinElmer Slide Holder H.45 (up to 4 Slides), Sample below Slide</Description><Well ID="Well:0" Row="1" Column="1"><WellSample ID="WellSample:0:1" Index="1" PositionX="900.911" PositionXUnit="µm" PositionY="-15916.0885" PositionYUnit="µm"><ImageRef ID="Image:0" /></WellSample></Well></Plate><Instrument ID="Instrument:0"><Microscope Manufacturer="PerkinElmer" Model="Phenix" /><Detector ID="Detector:0:1" Model="AndorZylaCam" /><Detector ID="Detector:0:2" Model="AndorZylaCam" /><Detector ID="Detector:0:3" Model="AndorZylaCam" /><Detector ID="Detector:0:4" Model="AndorZylaCam" /><Detector ID="Detector:0:5" Model="AndorZylaCam" /><Objective ID="Objective:0" NominalMagnification="40" LensNA="1.1" WorkingDistanceUnit="mm" /></Instrument><Image ID="Image:0"><AcquisitionDate>2021-04-28T11:00:11.1544884+01:00</AcquisitionDate><InstrumentRef ID="Instrument:0" /><ObjectiveSettings ID="Objective:0" /><Pixels ID="Pixels:0" DimensionOrder="XYZCT" Type="uint16" BigEndian="false" Interleaved="false" SizeX="35033" SizeY="30924" SizeC="5" SizeZ="1" SizeT="1" PhysicalSizeX="0.146165685373" PhysicalSizeY="0.146165685373"><Channel ID="Channel:0:1" Color="6750207" Name="DAPI" SamplesPerPixel="1" EmissionWavelength="456" ExcitationWavelength="375" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:1" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:2" Color="16711935" Name="Opal 520" SamplesPerPixel="1" EmissionWavelength="522" ExcitationWavelength="488" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:2" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:3" Color="12648703" Name="Opal 570" SamplesPerPixel="1" EmissionWavelength="599" ExcitationWavelength="561" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:3" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:4" Color="4539903" Name="Opal 650" SamplesPerPixel="1" EmissionWavelength="706" ExcitationWavelength="640" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:4" Binning="1x1" /><LightPath /></Channel><Channel ID="Channel:0:5" Color="14614527" Name="Atto 425" SamplesPerPixel="1" EmissionWavelength="482" ExcitationWavelength="425" ContrastMethod="Fluorescence" IlluminationType="Epifluorescence"><DetectorSettings ID="Detector:0:5" Binning="1x1" /><LightPath /></Channel><TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="1" FirstT="0" FirstZ="0" IFD="1" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="2" FirstT="0" FirstZ="0" IFD="2" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="3" FirstT="0" FirstZ="0" IFD="3" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><TiffData FirstC="4" FirstT="0" FirstZ="0" IFD="4" PlaneCount="1"><UUID FileName="A1_F1T0_max.ome.tiff">urn:uuid:2c89ff08-89cd-4e8a-81a9-aed4f263f832</UUID></TiffData><Plane ExposureTime="0.08" TheC="0" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="1" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="2" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="3" TheZ="0" TheT="0" /><Plane ExposureTime="0.1" TheC="4" TheZ="0" TheT="0" /></Pixels></Image><StructuredAnnotations><MapAnnotation ID="Annotation:Resolution:0" Namespace="openmicroscopy.org/PyramidResolution"><Value><M K="1">17516 15462</M><M K="2">8758 7731</M><M K="3">4379 3865</M><M K="4">2189 1932</M><M K="5">1094 966</M><M K="6">547 483</M></Value></MapAnnotation></StructuredAnnotations></OME>
  Software: PerkinElmer Acapella 5.3.0.127111
TIFF Directory at offset 0x6b75516a (1802850666)
  Image Width: 35033 Image Length: 30924
  Tile Width: 512 Tile Length: 512
  Resolution: 68416.3, 68416.3 pixels/cm
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: Deflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 14023
  Planar Configuration: single image plane
  SubIFD Offsets: 1988110042 2035012596 2047131488 2050800184 2051571778 2051767704
  ImageDescription:
  Software: PerkinElmer Acapella 5.3.0.127111
TIFF Directory at offset 0xa1e6b982 (2716252546)
  Image Width: 35033 Image Length: 30924
  Tile Width: 512 Tile Length: 512
  Resolution: 68416.3, 68416.3 pixels/cm
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: Deflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 8619
  Planar Configuration: single image plane
  SubIFD Offsets: 2882548494 2924261964 2935157616 2938525238 2939217978 2939391502
  ImageDescription:
  Software: PerkinElmer Acapella 5.3.0.127111
TIFF Directory at offset 0xd61a4cc2 (3592047810)
  Image Width: 35033 Image Length: 30924
  Tile Width: 512 Tile Length: 512
  Resolution: 68416.3, 68416.3 pixels/cm
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: Deflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 30884
  Planar Configuration: single image plane
  SubIFD Offsets: 3754266484 3794626162 3805032388 3808192808 3808848430 3809014032
  ImageDescription:
  Software: PerkinElmer Acapella 5.3.0.127111
TIFF Directory at offset 0x10526a7e8 (4381386728)
  Image Width: 35033 Image Length: 30924
  Tile Width: 512 Tile Length: 512
  Resolution: 68416.3, 68416.3 pixels/cm
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: Deflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 2376
  Planar Configuration: single image plane
  SubIFD Offsets: 4520457536 4553757060 4562067446 4564555308 4565070524 4565199660
  ImageDescription:
  Software: PerkinElmer Acapella 5.3.0.127111
melissalinkert commented 3 years ago

Thank you for the additional info. I can now reproduce this error message:

Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.pyramid.PyramidFromDirectoryWriter@4a87761d): java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at picocli.CommandLine.executeUserObject(CommandLine.java:1792)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2349)
    at picocli.CommandLine.parseWithHandler(CommandLine.java:2284)
    at picocli.CommandLine.call(CommandLine.java:2560)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.main(PyramidFromDirectoryWriter.java:210)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.initialize(PyramidFromDirectoryWriter.java:726)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:225)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:97)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
    ... 9 more

when running raw2ometiff test test-output.ome.tiff after bioformats2raw test.ome.tiff test --resolutions 5. However, I can't reproduce the error when running the same raw2ometiff command after bioformats2raw test.ome.tiff test --resolutions 5 --no-hcs. Can you try running bioformats2raw with --no-hcs again with a non-existent output directory, verify that raw2ometiff shows the same error, and send the complete output of ls -Rhl on the Zarr directory written by bioformats2raw?

The error occurs without the --no-hcs flag because the Well in the original OME-XML has an incorrect Row (and possibly Column) index:

<Plate ID="Plate:0" Name="RV_GON_01A5" ColumnNamingConvention="letter" RowNamingConvention="number" Columns="4" Rows="1" ExternalIdentifier="6f8a58d4-5b1e-4ecc-93fd-824a8d2c3468">
    <Description>PerkinElmer Slide Holder H.45 (up to 4 Slides), Sample below Slide</Description>
    <Well ID="Well:0" Row="1" Column="1">
      <WellSample ID="WellSample:0:1" Index="1" PositionX="900.911" PositionXUnit="µm" PositionY="-15916.0885" PositionYUnit="µm">
        <ImageRef ID="Image:0"/>
      </WellSample>
    </Well>
  </Plate>

For reference, I created the test input file by pasting the OME-XML above into a file 62.ome.xml, and then running:

$ bfconvert "test&sizeC=5&pixelType=uint16&sizeX=35033&sizeY=30924.fake" test.ome.tiff -pyramid-resolutions 7 -tilex 512 -tiley 512 -compression LZW
$ tiffcomment -set 62.ome.xml test.ome.tiff

This warning isn't related, and is fine to ignore:

OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp6105402081124591895/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix t
he stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
BioinfoTongLI commented 3 years ago

I added --no-hcs. Still the same error. and the ls -Rhl output is attached here: dir.txt

melissalinkert commented 3 years ago

I don't see anything obviously wrong with the ls -Rhl output. One final thing to check - can you please paste the contents of OME/METADATA.ome.xml from the --no-hcs run?

BioinfoTongLI commented 3 years ago

Sure. Just changed the suffix to txt (as github doesn't like .xml). METADATA.ome.txt

BioinfoTongLI commented 3 years ago

@melissalinkert Just FYI, I just tried the latest rc4 of bioformats2raw. Unfortunately, still the same error :( Lmk if there's anything else I can help

BioinfoTongLI commented 3 years ago

So what I can tell is that the same file can be converted using 0.2.6 correctly (both bioformats2raw and raw2ometiff). I've also tried bioformats2raw-0.2.6 with file-type zarr + raw2ometiff-0.3.0-rc3, and got this error:

  Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.pyramid.PyramidFromDirectoryWriter@8646db9): java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
    at picocli.CommandLine.executeUserObject(CommandLine.java:1792)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2349)
  Causedat picocli.CommandLine.parseWithHandler(CommandLine.java:2284)
    at picocli.CommandLine.call(CommandLine.java:2560)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.main(PyramidFromDirectoryWriter.java:210)
  Caused by: java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:239)
  Causedat com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:97)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
    ... 9 more
  Caused by: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
    at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:104)
    at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
    at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:87)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.createReader(PyramidFromDirectoryWriter.java:1150)
    at com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.initialize(PyramidFromDirectoryWriter.java:576)
  rk dirat com.glencoesoftware.pyramid.PyramidFromDirectoryWriter.call(PyramidFromDirectoryWriter.java:225)
  /home/... 11 more

I assume this is expected though.

melissalinkert commented 3 years ago

Thanks for your patience in debugging this. It's definitely expected that mixing 0.2.6 and 0.3.0rc builds won't work, but good to know that 0.2.6 works.

So far, I have only been able to reproduce the same error by doing the following (with 0.3.0rc*):

$ bioformats2raw --no-hcs test.ome.tiff test-no-hcs
$ raw2ometiff test-no-hcs/0/ test-output.ome.tiff

I know it's unlikely, but just to rule out the easy solution can you double-check the exact input path in the raw2ometiff command and make sure it's identical to the output path in the bioformats2raw command?

If that doesn't solve the problem, https://github.com/glencoesoftware/bioformats2raw/pull/102 and https://github.com/glencoesoftware/raw2ometiff/pull/63 add debug logging around the places that could be causing this. Builds with those changes are here:

https://ci.appveyor.com/api/buildjobs/ovjcjudk3o399lw0/artifacts/build%2Fdistributions%2Fbioformats2raw-0.3.0-SNAPSHOT.zip https://ci.appveyor.com/api/buildjobs/em3k3394d1ig4etw/artifacts/build%2Fdistributions%2Fraw2ometiff-0.3.0-SNAPSHOT.zip

If you can re-run bioformats2raw --no-hcs --debug and raw2ometiff --debug with those builds and paste the complete output of both commands here, that should hopefully provide some clues.

BioinfoTongLI commented 3 years ago

It should be me thanking you guys for this great tool. Just trying to help as more as I can.

As for the path, I can confirm that it's the correct path. I delete the directory with rm -rf for each test.

Just a question about debuging, which level should I use? The ALL gives a file of 33M and stopped by this error in bioformats2raw:

  Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@544a2ea6): java.lang.NullPointerException
    at picocli.CommandLine.executeUserObject(CommandLine.java:1792)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2349)
    at picocli.CommandLine.parseWithHandler(CommandLine.java:2284)
    at picocli.CommandLine.call(CommandLine.java:2560)
    at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:1613)
  Caused by: java.lang.NullPointerException
    at loci.common.RandomAccessInputStream.getFilePointer(RandomAccessInputStream.java:236)
    at loci.formats.tiff.OnDemandLongArray.toArray(OnDemandLongArray.java:79)
    at loci.formats.tiff.IFD.printIFD(IFD.java:980)
    at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:900)
    at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:869)
    at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:349)
    at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
    at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
    at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:161)
    at com.glencoesoftware.bioformats2raw.Converter.getTile(Converter.java:863)
    at com.glencoesoftware.bioformats2raw.Converter.processTile(Converter.java:959)
    at com.glencoesoftware.bioformats2raw.Converter.lambda$saveResolutions$4(Converter.java:1109)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
melissalinkert commented 3 years ago

The DEBUG level is fine for both commands.

BioinfoTongLI commented 3 years ago

Sorry was busy fixing microscope 🤣 Here are the logs for bf2raw and raw2ometiff bf2raw.log raw2tif.log

FYI, bf2raw threw no error. It crashed at raw2ometiff.

chris-allan commented 3 years ago

In the latest log file we have:

...
2021-05-12 16:02:41,947 [main] DEBUG c.g.p.PyramidFromDirectoryWriter -   group key count = 0
...

This is fatal. The implementation of key count retrieval is to walk the root group directory structure looking for .zgroup files. This returning 0 suggests either:

  1. The .zgroup files are missing (we'd need to look at ls -Rhla rather than ls -Rhl to check this)
  2. For some reason the .zgroup files cannot be read or seen by the directory walk

At the top of the bioformats2raw log I'm seeing bind mount warnings. What is the execution environment here? What type of local or remote filesystem is at your destination?

BioinfoTongLI commented 3 years ago

Thanks @chris-allan for looking at this! The message is very useful! I think the issue is partly related to the symbolic link. I will elaborate more below.

I am using nextflow. The zarr intermediare image is the output of the first process that wrapped bioformats2raw. The zarr is then copied to a nfs folder. I just check manually. The .zgroup are there, for both output of bioformats2raw and input of raw2ometiff. However, nextflow by default use a empty folder for each process. So for the step of raw2ometiff, it just stageIn the zarr using symbolic link (more details here: https://www.nextflow.io/docs/latest/process.html?highlight=stage#stageinmode). When I changed that to copy. It works of some of the images (none of the images worked before with this 0.3.0-rcx). So that's an improvment. However, some others still fail... I'll update more here when I find the solution.

I am sorry about this because as it's kind of my bad, as melissa already suggested to double check if the files exists so that raw2ometiff can see them. I didn't realize symlink is an issue as it worked with previous version.

chris-allan commented 3 years ago

Thanks, @BioinfoTongLI.

It sounds to me like we have a legitimate issue then with symlink to path resolution with raw2ometiff. Can you let me know what the symlink and/directory structure looks like on the second nextflow step so I can try to build a test case?

BioinfoTongLI commented 3 years ago

Sure, the top level looks like this:

  total 24K
  drwxrwxr-x 3 ubuntu ubuntu 4.0K Jun 23 17:25 ..
  drwxrwxr-x 9 ubuntu ubuntu 4.0K Jun 23 17:25 0
  drwxrwxr-x 4 ubuntu ubuntu 4.0K Jun 23 17:16 .
  -rw-rw-r-- 1 ubuntu ubuntu   33 Jun 23 17:16 .zattrs
  -rw-rw-r-- 1 ubuntu ubuntu   23 Jun 23 17:16 .zgroup
  drwxrwxr-x 2 ubuntu ubuntu 4.0K Jun 23 17:16 OME

The full zarr dir looks like this. Ran with ls -lhtaR symlink_issue 2.txt

chris-allan commented 3 years ago

Understood. I'd need to see the symlink though and what it points to.

BioinfoTongLI commented 3 years ago

Not sure I get 100% what you mean. but will this be helpful?

work/9f/b46c2e63a1735eeb96ec43adaa6b54/ is the isolated workspace.

and /home/ubuntu/Documents/nf_module_image_convert/test/raws/NC_WEM TOslide 15.6.21 - 2021-06-15 10.26.32.ome the output of bioformats2raw (the content is show in the last message)

❯ ls -ltha work/9f/b46c2e63a1735eeb96ec43adaa6b54/
total 36K
drwxrwxr-x 2 ubuntu ubuntu 4.0K Jun 23 19:36  .
-rw-rw-r-- 1 ubuntu ubuntu 2.4K Jun 23 19:36  .command.log
-rw-rw-r-- 1 ubuntu ubuntu    1 Jun 23 19:36  .exitcode
-rw-rw-r-- 1 ubuntu ubuntu 1.8K Jun 23 19:36  .command.err
-rw-rw-r-- 1 ubuntu ubuntu    0 Jun 23 19:36 'NC_WEM TOslide 15.6.21 - 2021-06-15 10.26.32.ome.tif'
-rw-rw-r-- 1 ubuntu ubuntu  307 Jun 23 19:36  .command.out
-rw-rw-r-- 1 ubuntu ubuntu    0 Jun 23 19:36  .command.begin
lrwxrwxrwx 1 ubuntu ubuntu  105 Jun 23 19:36 'NC_WEM TOslide 15.6.21 - 2021-06-15 10.26.32.ome' -> '/home/ubuntu/Documents/nf_module_image_convert/test/raws/NC_WEM TOslide 15.6.21 - 2021-06-15 10.26.32.ome'
-rw-rw-r-- 1 ubuntu ubuntu 3.7K Jun 23 19:36  .command.run
-rw-rw-r-- 1 ubuntu ubuntu  425 Jun 23 19:36  .command.sh
drwxrwxr-x 4 ubuntu ubuntu 4.0K Jun 23 19:36  ..

❯ pwd
/home/ubuntu/Documents/nf_module_image_convert
chris-allan commented 3 years ago

That's perfect, thanks.

chris-allan commented 3 years ago

Should be resolved by #65. 0.3.0-rc4 was released with that change included.