search

ome / ZarrReader

Other
7 stars 11 forks source link

Try to reproduce Well reordering issue with Fileset swapping #52

Open will-moore opened 1 year ago

will-moore commented 1 year ago

See e.g. https://github.com/IDR/idr-metadata/issues/652#issuecomment-1505229546 The reordering of Wells seen there is hard to work with since it needs .pattern files, which can't be easily imported alongside NGFF plates, since we don't have a server that can read both.

Try to test and use other HCS formats to reproduce the issue... Use small mias plate to test... - unfortunately, the converted NGFF plate failed to import...

$ rsync -rvP --progress idr0-slot3.openmicroscopy.org:/uod/idr/repos/curated/mias/frans/siRNA_PRIM1_03102008 .
$ bioformats2raw --version
Version = 0.4.0
Bio-Formats version = 6.8.0
$ bioformats2raw siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt plate.zarr
$ ls plate.zarr/
0   OME

Import original and NGFF plates... into merge-ci...

$ omero import siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt
...
2023-04-24 14:16:42,726 90075      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /Users/wmoore/Desktop/images/mias/siRNA_PRIM1_03102008/001-365700055641/Well0013/mode1_z000_t000_im1_1.tif
Plate:68655
Other imported objects:
Fileset:140757

$ omero import plate.zarr/OME/METADATA.ome.xml --depth=100
Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-04-24 14:19:06,331 467        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.12
2023-04-24 14:19:06,363 499        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0 revision: f8b46c2458c43cffdf5bc67cc4bf9dfc6e93167b date: 31 May 2022
2023-04-24 14:19:06,479 615        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-04-24 14:19:07,109 1245       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-04-24 14:20:01,084 55220      [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /Users/wmoore/Desktop/images/mias/plate.zarr/OME/METADATA.ome.xml
java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Integer (java.lang.String and java.lang.Integer are in module java.base of loader 'bootstrap')
    at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:539)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:287)
    at loci.formats.FormatReader.setId(FormatReader.java:1443)
    at loci.formats.ImageReader.setId(ImageReader.java:849)
    at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.Memoizer.setId(Memoizer.java:662)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
    at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
    at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
    at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
    at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
    at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
    at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:997)
2023-04-24 14:20:01,092 55228      [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 53966ms. (53983ms total) [0 unknowns]
2023-04-24 14:20:01,182 55318      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2023-04-24 14:20:01,851 55987      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-04-24 14:20:02,225 56361      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.5.12
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.14.1
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 12.2.1
No imports due to errors!

Also tried removing /OME dir and...

omero import plate.zarr/.zattrs --depth=100
No imports found
sbesson commented 1 year ago

Based on the stack trace, I suspect the issue is related to the acquisition.id metadata - which has been clarified in https://github.com/ome/ngff/pull/120, implemented in https://github.com/glencoesoftware/bioformats2raw/pull/152 and released in bioformats2raw 0.5.0. It would be useful to test whether the same plate converted with a recent version of bioformats2raw displays the same issue

will-moore commented 1 year ago 
$ conda update -c ome bioformats2raw
$ bioformats2raw --version
Version = 0.6.1
Bio-Formats version = 6.12.0
NGFF specification version = 0.4
will-moore commented 1 year ago

Want to try and setup a new env on idr0125-pilot for doing the conversion and import to that server, which has recent bf2raw fixes etc.

Created new env on idr0125-pilot, install bioformats2raw with $ conda install -c ome bioformats2raw :

$ bioformats2raw --version
Version = 0.6.1
Bio-Formats version = 6.12.0
NGFF specification version = 0.4

But can't get the data there...

(bioformats2raw) [wmoore@pilot-idr0125-omeroreadwrite mias]$ rsync -rvP --progress idr0-slot3.openmicroscopy.org:/uod/idr/repos/curated/mias/frans/siRNA_PRIM1_03102008 .
ssh: Could not resolve hostname idr0-slot3.openmicroscopy.org: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]

Since I have a local copy, upload to idr-pilot...

$ rsync -rvP --progress siRNA_PRIM1_03102008 idr-pilot.openmicroscopy.org:/home/wmoore/

Then copy from there to idr0125-pilot:

$ ssh -A idr-pilot.openmicroscopy.org
$ rsync -rvP --progress siRNA_PRIM1_03102008 idr0125-omeroreadwrite:/home/wmoore/
$ ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1080:localhost:80
$ mv siRNA_PRIM1_03102008 test_data/mias/
$ conda activate bioformats2raw
$ bioformats2raw siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt plate.zarr

cd ../
mv mias/ /ngff/
cd /ngff
sudo chown -R omero-server mias
$ sudo chown -R omero-server
$ source /opt/omero/server/venv3/bin/activate

$ omero import --transfer=ln_s --depth=100 siRNA_PRIM1_03102008/
Plate:10322
Other imported objects:
Fileset:5286923
==> Summary
139 files uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:01:31.977

(venv3) (venv3) (base) [omero-server@pilot-idr0125-omeroreadwrite mias]$ omero import --transfer=ln_s --depth=100 plate.zarr
Plate:10323
Other imported objects:
Fileset:5286924
==> Summary
358 files uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:06:17.050
will-moore commented 1 year ago 
$ python swap_filesets.py Plate:10322 Plate:10323
UPDATE pixels SET name = '.zattrs', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-04/24/16-49-56.080/plate.zarr' where image in (select id from Image where fileset = 5286924);

$ source /opt/omero/server/venv3/bin/activate
$ export OMERODIR=/opt/omero/server/OMERO.server
$ omero config get --show-password
$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102
UPDATE pixels SET name = '.zattrs', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-04/24/16-49-56.080/plate.zarr' where image in (select id from Image where fileset = 5286924);
UPDATE 16

Original Data:

Screenshot 2023-04-24 at 18 23 37

NGFF Plate:

Screenshot 2023-04-24 at 18 23 46

Original Plate after NGFF Fileswap and regenerate thumbnails (Save to All in webclient):

Screenshot 2023-04-24 at 18 27 32
will-moore commented 1 year ago 
sudo -u omero-server -s
cd
wget https://github.com/glencoesoftware/raw2ometiff/releases/download/v0.4.1/raw2ometiff-0.4.1.zip
unzip raw2ometiff-0.4.1.zip

cd /ngff/mias/
/opt/omero/server/raw2ometiff-0.4.1/bin/raw2ometiff plate.zarr plate.ome.tiff

source /opt/omero/server/venv3/bin/activate
export OMERODIR=/opt/omero/server/OMERO.server
omero import --transfer=ln_s  plate.ome.tiff 

Plate:10324
Other imported objects:
Fileset:5286925

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:03:58.218

Set the original imported Plate to use the ome.tiff Fileset...

python swap_filesets.py Plate:10322 Plate:10324

// psql - NB: needed to use plate.ome.tiff
idr=> UPDATE pixels SET name = 'plate.ome.tiff', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-14/2023-04/24/19-12-28.091/' where image in (select id from Image where fileset = 5286925);
UPDATE 16

This fixes the Wells ordering in the Original imported Plate, now using ome.tiff Fileset:

Screenshot 2023-04-24 at 20 32 14
sbesson commented 1 year ago

Thanks @will-moore. This seems to indicate 2 things: