search

TheJacksonLaboratory / ezomero

A module with convenience functions for writing Python code that interacts with OMERO.
GNU General Public License v2.0
39 stars 13 forks source link

[BUG] ZARR upload not working #109

Closed jo-mueller closed 3 months ago

jo-mueller commented 4 months ago

Describe the bug

I was trying to upload a .zarr image to omero with the most recent version and encountered an error message. Full traceback:

Using OMERO.java-5.6.11-ice36
Joined session for Johannes@172.26.122.141:4064. Idle timeout: 10 min. Current group: BiaPoL
ERROR:root:Import of [c:\...\notebooks\medium_image.zarr](file:///C:/.../notebooks/medium_image.zarr) has failed!
ERROR:root:File [c:\...\notebooks\medium_image.zarr](file:///C:/.../notebooks/medium_image.zarr) has not been imported
ERROR:root:No image ids to organize
WARNING:root:Missing annotation or namespace, skipping annotations
id_image None

To Reproduce

I essentially used this snippet to check whether the upload works. Full code:

import numpy as np
import zarr
import ezomero

from ome_zarr.io import parse_url
from ome_zarr.writer import write_image

# from https://forum.image.sc/t/images-batch-import-into-a-dataset-in-omero-using-ezomero-2-1-0/89529/12

def test_import(conn):

    # generate numpy data...
    # shape = (1, 4, 15, 9247, 8332)
    shape = (1, 2, 3, 1500, 3000)
    mean_val=10
    rng = np.random.default_rng(0)
    data = rng.poisson(mean_val, size=shape).astype(np.uint8)
    print(data.shape)

    # # write to ome-zarr...
    filename = "medium_image.zarr"
    chunks = (1, 1, 1, 1024, 1024)
    store = parse_url(filename, mode="w").store
    root = zarr.group(store=store)
    write_image(image=data, group=root, axes="tczyx", storage_options=dict(chunks=chunks))

    # import...
    id_image = ezomero.ezimport(conn=conn, target=filename)
    print('id_image', id_image)

if __name__ == '__main__':
    conn = ezomero.connect(host='my_omero_ip', port=4064, user='Johannes', group='my_group', secure=True)
    test_import(conn)

Expected behavior The uploaded image should upload to the omero server with a new id.

Desktop (please complete the following information):

Further context

It seems related to this issue on image.sc but then again, the problem should have been fixed with the recent release 🤔

cc @zoccoler @cwetzker

erickmartins commented 3 months ago

I'm trying to reproduce this locally, but having trouble with the code snippet there (it fails at writing the zarr itself). I'd still expect that something like ezomero.ezimport(conn=conn, target=filename, depth=10) is necessary to get zarr imports working properly - have you tried that?

jo-mueller commented 3 months ago

Hi @erickmartins ,

sorry for the late reply - I only got around to testing just now. This is an updated snippet that hopefully reproduces the error. Requires pip install bioio bioio-ome-zarr.

from bioio.writers.ome_zarr_writer import OmeZarrWriter

writer = OmeZarrWriter('test.zarr')
writer.write_image(
    np.random.rand(1, 1, 64, 128, 128).astype(np.float32),
    chunk_dims=(1, 1, 64, 128, 128),
    scale_num_levels=1,
    dimension_order='TCZYX',
    image_name='test',
    channel_names=['test'],
    channel_colors=None,
    physical_pixel_sizes = None,
)

ezomero.ezimport(conn=conn, target='test.zarr', dataset=255, depth=10)

That's the traceback:

Using OMERO.java-5.6.10-ice36
Joined session for johamuel@omero-int.biotec.tu-dresden.de:4064. Idle timeout: 10 min. Current group: Dye Lab
2024-08-01 18:03:06,231 146        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.2
2024-08-01 18:03:06,240 155        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.1.0 revision: 05c7b2413cfad19a73b619c61ddf77ca2d038ce7 date: 11 December 2023
2024-08-01 18:03:06,270 185        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-08-01 18:03:06,452 367        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2024-08-01 18:03:06,562 477        [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /home/pol_haase/johamuel/Documents/Git/single-cell-analysis-of-organoids/notebooks/00_data_management/test.zarr/0/0/0/0/0/0
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
    at loci.formats.in.ZarrReader.parseOmeroMetadata(ZarrReader.java:704)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:170)
    at loci.formats.FormatReader.setId(FormatReader.java:1480)
    at loci.formats.ImageReader.setId(ImageReader.java:865)
    at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.ChannelFiller.setId(ChannelFiller.java:258)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:317)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.Memoizer.setId(Memoizer.java:698)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
    at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
...
2024-08-01 18:03:06,957 872        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 5.15.0-25-generic
No imports due to errors!
ERROR:root:No image ids to organize
WARNING:root:Missing annotation or namespace, skipping annotations
joshmoore commented 3 months ago

Can you share the test.zarr, @jo-mueller? (Or at least, the metadata files in the fileset)

cc: @dgault

jo-mueller commented 3 months ago

@joshmoore coming right up 👍

https://datashare.tu-dresden.de/s/2tA2rqLb7XjYBL5

joshmoore commented 3 months ago

Thanks! It certainly validates:

image

The line of code doesn't match the current reader though: https://github.com/ome/ZarrReader/blob/main/src/loci/formats/in/ZarrReader.java

Do you know what version you are using?

jo-mueller commented 3 months ago

Ezomero version is 3.0.1 if that's what you mean?

joshmoore commented 3 months ago

Ezomero version is 3.0.1 if that's what you mean?

Sorry, no. Of the ZarrReader.

erickmartins commented 3 months ago

yeah, this looks like an error at the omero import level, and not specific to ezomero - in fact, it might be worth seeing if a "naked" omero import works, and if it doesn't, this is probably an issue worth opening against the relevant OME repo (ZarrReader, I assume?).

jo-mueller commented 3 months ago

@joshmoore these are the versions I used, which I think are the latest ones

bioio 1.0.3
bioio-ome-zarr = 1.0.1
ome-zarr 0.9.0
omero-py 5.13.1
erickmartins commented 3 months ago

can you confirm if this works with omero import?

jo-mueller commented 3 months ago

CLI import command

omero import --name "Test" --depth 10 ./test.zarr/

and full traceback

Using OMERO.java-5.6.10-ice36
Using session for johamuel@omero-int.biotec.tu-dresden.de:4064. Idle timeout: 10 min. Current group: default
2024-08-02 15:33:43,029 145        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.2
2024-08-02 15:33:43,038 154        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.1.0 revision: 05c7b2413cfad19a73b619c61ddf77ca2d038ce7 date: 11 December 2023
2024-08-02 15:33:43,069 185        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-08-02 15:33:43,247 363        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2024-08-02 15:33:43,356 472        [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /home/pol_haase/johamuel/Documents/Git/single-cell-analysis-of-organoids/notebooks/00_data_management/./test.zarr/0/0/0/0/0/0
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
    at loci.formats.in.ZarrReader.parseOmeroMetadata(ZarrReader.java:704)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:170)
    at loci.formats.FormatReader.setId(FormatReader.java:1480)
    at loci.formats.ImageReader.setId(ImageReader.java:865)
    at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.ChannelFiller.setId(ChannelFiller.java:258)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:317)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at loci.formats.Memoizer.setId(Memoizer.java:698)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
    at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
    at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
    at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:355)
    at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:381)
    at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
    at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
    at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
    at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2024-08-02 15:33:43,358 474        [      main] INFO      ome.formats.importer.ImportCandidates - 4 file(s) parsed into 0 group(s) with 1 call(s) to setId in 109ms. (112ms total) [0 unknowns]
2024-08-02 15:33:43,391 507        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to omero-int.biotec.tu-dresden.de:4064
2024-08-02 15:33:43,593 709        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2024-08-02 15:33:43,758 874        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.2
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.24
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 5.15.0-25-generic

so yeah...issue may have to be moved upstream.

erickmartins commented 3 months ago

Yeah, it looks like that - @joshmoore I assume to ZarrReader?

joshmoore commented 3 months ago

Yes, though I'd suggest trying to upgrade first:

omero import --fetch-jars 5.6.12
Downloading https://downloads.openmicroscopy.org/omero/5.6.12/OMERO.java.zip

If that changes things, then you'll also need a server upgrade to have the ZarrReader on both sides. Otherwise, --> github.com/ome/ZarrReader

jo-mueller commented 3 months ago

Same error, so I guess that's where I'll turn next. Thanks for the help on this end!

jo-mueller commented 3 months ago

---> Follow-up here