search

ome / omero-cli-zarr

https://pypi.org/project/omero-cli-zarr/
GNU General Public License v2.0
15 stars 10 forks source link

Support export of Fileset:ID #123

Open will-moore opened 2 years ago

will-moore commented 2 years ago

Fixes #122

To test: e.g. on merge-ci (user-3):

$ omero zarr export Fileset:115357 --output="test_fileset"

Will create a test_fileset directory with images /0, /1 etc in it. Directory will contain .zattrs and /OME/METADATA.ome.xml as defined in https://github.com/ome/ngff/pull/112

If directory exists, then test_fileset(1) directory will be created etc.

Support for other multi-image export to the same format NOT included yet. Could come in a follow-up PR e.g:

$ omero zarr export Image:10,11,12 --output="my_images"
$ omero zarr export Dataset:123 --output="my_images"

(would need different object parameter type since current ProxyStringType doesn't support multiple IDs).

NB: to get the Fileset ID from an Image in webclient, see https://github.com/ome/omero-web/pull/385

will-moore commented 2 years ago

Webclient PR to show Fileset ID: https://github.com/ome/omero-web/pull/385

joshmoore commented 2 years ago

@pwalczysko, but do you also want to give this PR a try? :wink:

pwalczysko commented 2 years ago 
(cli) pwalczysko@ls31618~/Work/omero-cli-zarr$ pip install .
Processing /Users/pwalczysko/Work/omero-cli-zarr
Requirement already satisfied: omero-py>=5.6.0 in /Users/pwalczysko/opt/anaconda3/envs/cli/lib/python3.6/site-packages (from omero-cli-zarr==0.5.1.dev4+g553b35f) (5.9.1)
Collecting ome-zarr>=0.5.0
  Using cached ome_zarr-0.5.2-py3-none-any.whl (32 kB)
ERROR: Could not find a version that satisfies the requirement ome-types (from omero-cli-zarr)
ERROR: No matching distribution found for ome-types
will-moore commented 2 years ago

I couldn't reproduce that error. If I created a clean env using the omero-py docs conda create -n myenv -c conda-forge python=3.8 omero-py then installed omero-cli-zarr with pip install . this worked fine. If I removed the omero-py from the conda create command then installing omero-cli-zarr fails with ERROR: Failed building wheel for zeroc-ice, which seems to be unrelated to this PR (I'll create another issue)...

But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?

pwalczysko commented 2 years ago

I couldn't reproduce that error. If I created a clean env using the omero-py docs conda create -n myenv -c conda-forge python=3.8 omero-py then installed omero-cli-zarr with pip install . this worked fine. If I removed the omero-py from the conda create command then installing omero-cli-zarr fails with ERROR: Failed building wheel for zeroc-ice, which seems to be unrelated to this PR (I'll create another issue)...

But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?

I think that the discrepancy is in the python version, my envs have python 3.6. Will try

pwalczysko commented 2 years ago

Running a conda create -n myenv -c conda-forge python=3.8 #leaving out the omeropy bit conda activate myenv pip install .

gives me a following error @will-moore

``` ... clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/pwalczysko/opt/anaconda3/envs/testzarr/include -fPIC -O2 -isystem /Users/pwalczysko/opt/anaconda3/envs/testzarr/include -arch x86_64 -DICE_STATIC_LIBS -Isrc -Isrc/ice/cpp/include -Isrc/ice/cpp/src -I/Users/pwalczysko/opt/anaconda3/envs/testzarr/include/python3.8 -c src/ice/cpp/src/IceSSL/AcceptorI.cpp -o build/temp.macosx-10.9-x86_64-cpython-38/src/ice/cpp/src/IceSSL/AcceptorI.o -w In file included from src/ice/cpp/src/IceSSL/AcceptorI.cpp:14: In file included from src/ice/cpp/src/IceSSL/OpenSSLTransceiverI.h:14: In file included from src/ice/cpp/src/IceSSL/Util.h:23: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:27: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecBase.h:88:33: error: typedef redefinition with different types ('struct __SecCertificate' vs 'OpaqueSecCertificateRef') typedef struct __SecCertificate OpaqueSecCertificateRef; ^ src/ice/cpp/include/IceSSL/Plugin.h:67:8: note: previous definition is here struct OpaqueSecCertificateRef; ^ In file included from src/ice/cpp/src/IceSSL/AcceptorI.cpp:14: In file included from src/ice/cpp/src/IceSSL/OpenSSLTransceiverI.h:14: In file included from src/ice/cpp/src/IceSSL/Util.h:23: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:27: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecBase.h:111:25: error: typedef redefinition with different types ('struct __SecKey' vs 'OpaqueSecKeyRef') typedef struct __SecKey OpaqueSecKeyRef; ^ src/ice/cpp/include/IceSSL/Plugin.h:73:8: note: previous definition is here struct OpaqueSecKeyRef; ^ 2 errors generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> zeroc-ice note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. ```
will-moore commented 2 years ago

Issue created at https://github.com/ome/omero-cli-zarr/issues/127

will-moore commented 2 years ago

@pwalczysko I don't see ERROR: No matching distribution found for ome-types there? Do you know how you got that (since that seems more relevant to this PR)?

pwalczysko commented 2 years ago

@pwalczysko I don't see ERROR: No matching distribution found for ome-types there? Do you know how you got that (since that seems more relevant to this PR)?

conda create -n test2 python=3.6
conda activate test2
pip install .

elicits the error below @will-moore (note python version in the above command)

``` Processing /Users/pwalczysko/Work/omero-cli-zarr DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Collecting omero-py>=5.6.0 Using cached omero_py-5.11.2-py3-none-any.whl (2.8 MB) Collecting ome-zarr>=0.5.0 Using cached ome_zarr-0.5.2-py3-none-any.whl (32 kB) ERROR: Could not find a version that satisfies the requirement ome-types (from omero-cli-zarr) (from versions: none) ERROR: No matching distribution found for ome-types ```
will-moore commented 2 years ago

Ah - OK, thanks. So the use of python-3.6 should also be handled by better install instructions: https://github.com/ome/omero-cli-zarr/issues/127

If you install omero-py first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?

pwalczysko commented 2 years ago

If you install omero-py first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?

Yes, this is what I did and it works.

pwalczysko commented 2 years ago

Is this functionality supposed to produce real ome.zarr images with pixeldata ?

See https://merge-ci.openmicroscopy.org/web/webclient/?show=image-243571 - user-3, after reimport into OMERO, all values reported as 0, the original image is the https://merge-ci.openmicroscopy.org/web/webclient/?show=image-200515

The same is repeating with any image I tried up till now.

will-moore commented 2 years ago

@pwalczysko yes, it is supposed to!

omero zarr export Fileset:115357 --output="test_reimport"

// reimport using latest OMERO release
echo $OMERODIR 
/Users/wmoore/Desktop/OMERO.server-5.6.5-ice36-b233/

omero import --depth=10 test_reimport/
...
Image:243573,243574,243575
Other imported objects:
Fileset:128463

==> Summary
1 file uploaded, 1 fileset created, 3 images imported, 0 errors in 0:00:03.459

It looks like the data chunks haven't been uploaded. And indeed, these images appear to have all pixel values of 0.

Although they appear OK in napari:

napari --plugin napari-ome-zarr test_reimport/0/

Trying to export and re-import a single image:

$ omero zarr export Image:206965

$ omero import --depth=10 206965.zarr
...
// I see upload of all chunks etc...

2022-08-03 13:22:11,446 16144      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488889
2022-08-03 13:22:11,641 16339      [l.Client-0] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/206965.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
    at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.hasAsKey(JacksonAnnotationIntrospector.java:1080)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:495)

==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.060

Trying to import a single image from the Fileset exported above...

omero import --depth=10 test_reimport/0
...
No imports found

Seems that Bio-Formats doesn't recognise this as an OME-NGFF image without .zarr... Try to rename... get same error as above...

mv test_reimport/0 test_reimport/0.zarr
omero import --depth=10 test_reimport/0.zarr/

2022-08-03 13:32:18,262 16364      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488933
2022-08-03 13:32:18,459 16561      [  Thread-7] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/test_reimport/0.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey

==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.335

To summarise, import either fails to import the chunks (and you end up with all pixel values of 0) or if chunks are uploaded then the import fails for another reason.

cc @dgault any ideas what's going on here?

pwalczysko commented 2 years ago

Thank you @will-moore for the explanation and follow-up testing. From my side, the functionality in this PR works as described, but the reimport of the ome.zarr into OMERO fails as detailed in https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203891621 - I think this PR is okay, but maybe we should wait with the merging to find out what must be done to fix the complete export -> reimport workflow ?

will-moore commented 2 years ago

Trying to import the .zattrs file directly (as suggested by @pwalczysko ):

$ omero import --depth=10 test_reimport/0.zarr/.zattrs
No imports found
pwalczysko commented 2 years ago 
(.venv3) bash-4.2$ omero import -f  --depth 10 test_real/
2022-08-03 13:32:12,471 201        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.13-SNAPSHOT
2022-08-03 13:32:12,488 218        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.1-SNAPSHOT revision: f45c900610fd6a1c463ec4e21f45d809a38981b6 date: 3 August 2022
2022-08-03 13:32:12,528 258        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-08-03 13:32:12,863 593        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-08-03 13:32:28,537 16267      [      main] INFO      ome.formats.importer.ImportCandidates - 1028 file(s) parsed into 1 group(s) with 1021 call(s) to setId in 14209ms. (15674ms total) [1020 unknowns]
#======================================
# Group: /uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.OMEXMLReader
/uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml

@will-moore @dgault further to https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203891621 - I do not think that the correct reader is used, see above.

joshmoore commented 2 years ago

@pwalczysko https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203841319 Yes, this is what I did and it works.

(Side note: internally and externally we should always start from conda create -n test2 -c conda-forge omero-py in order to correctly install Ice)

imagesc-bot commented 1 year ago

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/export-large-images-from-omero-via-web/84652/3

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#343. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#344. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#345. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#346. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#347. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 8 months ago

Conflicting PR. Removed from build OMERO-plugins-push#348. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#349. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#350. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#351. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#352. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#353. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#354. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#355. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#356. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#357. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#358. See the console output for more details. Possible conflicts:

--conflicts

snoopycrimecop commented 7 months ago

Conflicting PR. Removed from build OMERO-plugins-push#359. See the console output for more details. Possible conflicts:

--conflicts