Open will-moore opened 2 years ago
Webclient PR to show Fileset ID: https://github.com/ome/omero-web/pull/385
@pwalczysko, but do you also want to give this PR a try? :wink:
(cli) pwalczysko@ls31618~/Work/omero-cli-zarr$ pip install .
Processing /Users/pwalczysko/Work/omero-cli-zarr
Requirement already satisfied: omero-py>=5.6.0 in /Users/pwalczysko/opt/anaconda3/envs/cli/lib/python3.6/site-packages (from omero-cli-zarr==0.5.1.dev4+g553b35f) (5.9.1)
Collecting ome-zarr>=0.5.0
Using cached ome_zarr-0.5.2-py3-none-any.whl (32 kB)
ERROR: Could not find a version that satisfies the requirement ome-types (from omero-cli-zarr)
ERROR: No matching distribution found for ome-types
I couldn't reproduce that error.
If I created a clean env using the omero-py
docs conda create -n myenv -c conda-forge python=3.8 omero-py
then installed omero-cli-zarr with pip install .
this worked fine.
If I removed the omero-py
from the conda create command then installing omero-cli-zarr fails with ERROR: Failed building wheel for zeroc-ice
, which seems to be unrelated to this PR (I'll create another issue)...
But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?
I couldn't reproduce that error. If I created a clean env using the
omero-py
docsconda create -n myenv -c conda-forge python=3.8 omero-py
then installed omero-cli-zarr withpip install .
this worked fine. If I removed theomero-py
from the conda create command then installing omero-cli-zarr fails withERROR: Failed building wheel for zeroc-ice
, which seems to be unrelated to this PR (I'll create another issue)...But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?
I think that the discrepancy is in the python version, my envs have python 3.6. Will try
Running a
conda create -n myenv -c conda-forge python=3.8 #leaving out the omeropy bit
conda activate myenv
pip install .
gives me a following error @will-moore
Issue created at https://github.com/ome/omero-cli-zarr/issues/127
@pwalczysko I don't see ERROR: No matching distribution found for ome-types
there? Do you know how you got that (since that seems more relevant to this PR)?
@pwalczysko I don't see
ERROR: No matching distribution found for ome-types
there? Do you know how you got that (since that seems more relevant to this PR)?
conda create -n test2 python=3.6
conda activate test2
pip install .
elicits the error below @will-moore (note python version in the above command)
Ah - OK, thanks. So the use of python-3.6 should also be handled by better install instructions: https://github.com/ome/omero-cli-zarr/issues/127
If you install omero-py
first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?
If you install omero-py first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?
Yes, this is what I did and it works.
Is this functionality supposed to produce real ome.zarr images with pixeldata ?
See https://merge-ci.openmicroscopy.org/web/webclient/?show=image-243571 - user-3, after reimport into OMERO, all values reported as 0, the original image is the https://merge-ci.openmicroscopy.org/web/webclient/?show=image-200515
The same is repeating with any image I tried up till now.
@pwalczysko yes, it is supposed to!
omero zarr export Fileset:115357 --output="test_reimport"
// reimport using latest OMERO release
echo $OMERODIR
/Users/wmoore/Desktop/OMERO.server-5.6.5-ice36-b233/
omero import --depth=10 test_reimport/
...
Image:243573,243574,243575
Other imported objects:
Fileset:128463
==> Summary
1 file uploaded, 1 fileset created, 3 images imported, 0 errors in 0:00:03.459
It looks like the data chunks haven't been uploaded. And indeed, these images appear to have all pixel values of 0.
Although they appear OK in napari
:
napari --plugin napari-ome-zarr test_reimport/0/
Trying to export and re-import a single image:
$ omero zarr export Image:206965
$ omero import --depth=10 206965.zarr
...
// I see upload of all chunks etc...
2022-08-03 13:22:11,446 16144 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488889
2022-08-03 13:22:11,641 16339 [l.Client-0] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/206965.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.hasAsKey(JacksonAnnotationIntrospector.java:1080)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:495)
==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.060
Trying to import a single image from the Fileset exported above...
omero import --depth=10 test_reimport/0
...
No imports found
Seems that Bio-Formats doesn't recognise this as an OME-NGFF image without .zarr
...
Try to rename... get same error as above...
mv test_reimport/0 test_reimport/0.zarr
omero import --depth=10 test_reimport/0.zarr/
2022-08-03 13:32:18,262 16364 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488933
2022-08-03 13:32:18,459 16561 [ Thread-7] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/test_reimport/0.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.335
To summarise, import either fails to import the chunks (and you end up with all pixel values of 0) or if chunks are uploaded then the import fails for another reason.
cc @dgault any ideas what's going on here?
Thank you @will-moore for the explanation and follow-up testing. From my side, the functionality in this PR works as described, but the reimport of the ome.zarr into OMERO fails as detailed in https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203891621 - I think this PR is okay, but maybe we should wait with the merging to find out what must be done to fix the complete export -> reimport workflow ?
Trying to import the .zattrs
file directly (as suggested by @pwalczysko ):
$ omero import --depth=10 test_reimport/0.zarr/.zattrs
No imports found
(.venv3) bash-4.2$ omero import -f --depth 10 test_real/
2022-08-03 13:32:12,471 201 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.13-SNAPSHOT
2022-08-03 13:32:12,488 218 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 6.10.1-SNAPSHOT revision: f45c900610fd6a1c463ec4e21f45d809a38981b6 date: 3 August 2022
2022-08-03 13:32:12,528 258 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-08-03 13:32:12,863 593 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-08-03 13:32:28,537 16267 [ main] INFO ome.formats.importer.ImportCandidates - 1028 file(s) parsed into 1 group(s) with 1021 call(s) to setId in 14209ms. (15674ms total) [1020 unknowns]
#======================================
# Group: /uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.OMEXMLReader
/uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml
@will-moore @dgault further to https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203891621 - I do not think that the correct reader is used, see above.
@pwalczysko https://github.com/ome/omero-cli-zarr/pull/123#issuecomment-1203841319 Yes, this is what I did and it works.
(Side note: internally and externally we should always start from conda create -n test2 -c conda-forge omero-py
in order to correctly install Ice)
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/export-large-images-from-omero-via-web/84652/3
Conflicting PR. Removed from build OMERO-plugins-push#343. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#344. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#345. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#346. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#347. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#348. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#349. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#350. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#351. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#352. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#353. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#354. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#355. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#356. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#357. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#358. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build OMERO-plugins-push#359. See the console output for more details. Possible conflicts:
--conflicts
Fixes #122
To test: e.g. on merge-ci (user-3):
Will create a
test_fileset
directory with images/0
,/1
etc in it. Directory will contain.zattrs
and/OME/METADATA.ome.xml
as defined in https://github.com/ome/ngff/pull/112If directory exists, then
test_fileset(1)
directory will be created etc.Support for other multi-image export to the same format NOT included yet. Could come in a follow-up PR e.g:
(would need different
object
parameter type since currentProxyStringType
doesn't support multiple IDs).NB: to get the Fileset ID from an Image in webclient, see https://github.com/ome/omero-web/pull/385