search

IDR / deployment

Deployment infrastructure for the Image Data Resource
https://idr.openmicroscopy.org/about/deployment.html
BSD 2-Clause "Simplified" License
13 stars 14 forks source link

OMERO.server 5.6.3 upgrade #327

Closed sbesson closed 4 months ago

sbesson commented 3 years ago

An initial attempt to upgrade the OMERO.server beyond 5.6.0 had been attempted in https://github.com/IDR/deployment/pull/229 but aborted and captured as https://github.com/ome/omero-server/issues/93. At minimum, the server changes invalidated the Bio-Formats cache files which required

As the latest OpenJDK releases drop support for TLS 1.0/1.1, imports will be broken in the next rolling deployment of IDR (pilot or production). The current CentOS 7 workaround is to set-up self-signed certificates server-side using omero-certificates - see https://github.com/ome/ansible-role-omero-server/issues/57.

In terms of deployment changes, the relevant line is:

diff --git a/ansible/group_vars/omero-hosts.yml b/ansible/group_vars/omero-hosts.yml
index f1e6763..dade77c 100644
--- a/ansible/group_vars/omero-hosts.yml
+++ b/ansible/group_vars/omero-hosts.yml
@@ -64,6 +64,8 @@ omero_server_datadir_bioformatscache: /data/BioFormatsCache

 omero_server_systemd_limit_nofile: 16384

+omero_server_selfsigned_certificates: True
+
 omero_server_python_addons:
 - omero-cli-render==0.7.0
 - omero-metadata==0.7.0

Testing this in a pilot environment with Java upgraded (via yum update) results in

(base) [sbesson@pilot-idr0072-omeroreadwrite ~]$  /opt/omero/server/OMERO.server/bin/omero import test.fake 
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
2021-04-27 10:06:27,638 270        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.5.5
2021-04-27 10:06:27,654 286        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.6.6 revision: 55bcf78bf629ab5dfc5f3e4f677043f11cbc4a57 date: 5 March 2021
2021-04-27 10:06:27,708 340        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2021-04-27 10:06:28,093 725        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2021-04-27 10:06:28,155 787        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 58ms. (62ms total) [0 unknowns]
2021-04-27 10:06:28,204 836        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2021-04-27 10:06:28,633 1265       [      main] ERROR  formats.importer.cli.CommandLineImporter - Error during import process.
Ice.SecurityException: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

From the error message, my understanding is that the relevant change is https://github.com/ome/omero-blitz/pull/108 i.e. a secondary requirement will be to upgrade the server to a recent version of OMERO.server containing omero-blitz 5.5.8 or later.

sbesson commented 3 years ago

@joshmoore suggested the exception mentioned above might only require a new version of omero-blitz client-side. Manually replacing omero-blitz-5.5.8.jar and omero-romio-5.6.2.jar under /opt/omero/server/OMERO.server/lib/client/ is indeed sufficient to let a minimal fake import.

This gives us a potential workaround in case we don't want to commit to a full server upgrade. However, I think further delaying this long-due upgrade and creating more custom deployments will eventually cost us. My vote would be for trying out the upgrade in a test environment potentially together with the micro-services deployment.

joshmoore commented 3 years ago

An initial attempt to upgrade the OMERO.server beyond 5.6.0 had been attempted in #229 but aborted and captured as ome/omero-server#93.

https://github.com/ome/omero-server/issues/93 almost certainly needs a PR against https://github.com/ome/omero-model/blob/master/src/main/resources/templates/psql-footer.vm#L782 and a release along with the full jar chain.

imagesc-bot commented 3 years ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/omero-import-error-ice-securityexception-javax-net-ssl-sslhandshakeexception-no-appropriate-protocol-again/52154/4

imagesc-bot commented 3 years ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/omero-import-error-ice-securityexception-javax-net-ssl-sslhandshakeexception-no-appropriate-protocol-again/52154/5

sbesson commented 3 years ago

Quick summary: https://github.com/IDR/deployment/pull/329, OMERO.server 5.6.3 has been deployed on a new instance of IDR without any DB change (no index creation). The old Bio-Formats cache files have been removed and various representative files have been tested across studies.

After regenerating the cache files for idr0044, retested the use case describe in https://github.com/ome/omero-model/issues/71 and the simple query returns instantly

[sbesson@prod97-omeroreadwrite ~]$ time /opt/omero/server/venv3/bin/python test.py 
(2048, 2169)

real    0m1.758s
user    0m1.571s
sys 0m1.660s

Additionally regenerated Bio-FOrmats cache files for several studies including idr0106 (OME-TIFF), idr0044 (KLB), idr0037, idr0081, idr0065, idr0092, idr0097, idr0095, idr0094, idr0045, idr0099, idr0079, idr0066. All these were successful.

A new blocker was revealed while trying to regenerate some cache files for .screen based studies (like idr0011). The reader initializes as expected but then fails with


2021-04-30 10:20:12,576 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-10) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328/metadata/idr0011-thorpe-Dad4/screens/Plate1-Blue-A.screen
2021-04-30 10:20:12,577 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328/metadata/idr0011-thorpe-Dad4/screens/Plate1-Blue-A.screen
java.lang.RuntimeException: ome.conditions.SecurityViolation: reader for /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328/metadata/idr0011-thorpe-Dad4/screens/Plate1-Blue-A.screen accesses data outside managed repository:
    /uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T37 x deletion library/Plate1/Plate1-Blue-A/P1-Bl-A-A1-1.zvi
    /uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T37 x deletion library/Plate1/Plate1-Blue-A/P1-Bl-A-A2-1.zvi
    /uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T37 x deletion library/Plate1/Plate1-Blue-A/P1-Bl-A-A3-1.zvi
    /uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T37 x deletion library/Plate1/Plate1-Blue-A/P1-Bl-A-A4-1.zvi
...

Also for idr0026

Caused by: ome.conditions.SecurityViolation: reader for /data/OMERO/ManagedRepository/demo_2/2017-04/12/13-14-13.681/tmp/idr-metadata/idr0026-weigelin-immunotherapy/patterns/3.50.6-3.140922_11-36-07.00.pattern accesses data outside managed repository:
    /uod/idr/filesets/idr0026-weigelin-immunotherapy/20170222-symlinks/PNAS_2015/treatment start day 3/mouse 50/day 6-3/time lapse/140922_11-36-07/11-36-07_PMT - PMT [FD6_GREEN] [00]_Time Time0003.tif
    /uod/idr/filesets/idr0026-weigelin-immunotherapy/20170222-symlinks/PNAS_2015/treatment start day 3/mouse 50/day 6-3/time lapse/140922_11-36-07/11-36-07_PMT - PMT [FD5_BLUE] [00]_Time Time0003.tif
    /uod/idr/filesets/idr0026-weigelin-immunotherapy/20170222-symlinks/PNAS_2015/treatment start day 3/mouse 50/day 6-3/time lapse/140922_11-36-07/11-36-07_PMT - PMT [BD8_RED] [00]_Time Time0003.tif
    /uod/idr/filesets/idr0026-weigelin-immunotherapy/20170222-symlinks/PNAS_2015/treatment start day 3/mouse 50/day 6-3/time lapse/140922_11-36-07/11-36-07_PMT - PMT [BD7_RED] [00]_Time Time0003.tif
    /uod/idr/filesets/idr0026-weigelin-immunotherapy/20170222-symlinks/PNAS_2015/treatment start day 3/mouse 50/day 6-3/time lapse/140922_11-36-07/11-36-07_PMT - PMT [FD6_GREEN] [00]_Time Time0004.tif
...

The key issue here is that the screen/pattern companion files used for creating the rich representation of these filesets points directly to the absolute path on disk:

https://github.com/IDR/idr-metadata/blob/954a7709b94f34b7ca7e21c631d9b10d1b0e31c4/idr0011-ledesmafernandez-dad4/screens/Plate1-Blue-A.screen#L2 https://github.com/IDR/idr-metadata/blob/954a7709b94f34b7ca7e21c631d9b10d1b0e31c4/idr0026-weigelin-immunotherapy/patterns/3.49.6-3.140922_11-33-57.00.pattern#L2

And this defies the current restrictions imposed by the 2016-SV1 where the data returned by getUsedFiles must be under the ManagedRepository.

Pattern-based studies where the data is contained e.g. by having the pattern on NFS like idr0045 or the data symlinked into the metadata repositories like idr0099 are regenerating without issue.

Investigating whether this can be worked around by updating the .screen/.pattern files to use relative links rather than absolute links.

sbesson commented 3 years ago

Shortlist of impacted studies (as returned by git grep -l /uod/idr/filesets | grep -v "\-plates.tsv" | grep -v "\-filePaths.tsv")

sbesson commented 3 years ago

Initial test while replacing the symlinks

[sbesson@prod97-omeroreadwrite 15-27-40.023]$ ls -alh
total 4.0K
drwxrwsr-x. 3 omero-server omero-server   44 Apr 30 13:51 .
drwxrwsr-x. 3 omero-server omero-server   48 Mar 15  2019 ..
drwxrwsr-x. 2 omero-server omero-server 4.0K Mar 15  2019 Tonsil 1
lrwxrwxrwx. 1 sbesson      idr-data       85 Mar 15  2019 Tonsil 1.pattern -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1.pattern
[sbesson@prod97-omeroreadwrite 15-27-40.023]$ ls -alh Tonsil\ 1
total 4.0K
drwxrwsr-x. 2 omero-server omero-server 4.0K Mar 15  2019 .
drwxrwsr-x. 3 omero-server omero-server   44 Apr 30 13:51 ..
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C00.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C00.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C01.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C01.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C02.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C02.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C03.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C03.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C04.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C04.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C05.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C05.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C06.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C06.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C07.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C07.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C08.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C08.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C09.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C09.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C10.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C10.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C11.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C11.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C12.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C12.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C13.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C13.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C14.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C14.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C15.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C15.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C16.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C16.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C17.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C17.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C18.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C18.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C19.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C19.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C20.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C20.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C21.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C21.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C22.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C22.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C23.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C23.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C24.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C24.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C25.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C25.png
lrwxrwxrwx. 1 sbesson      idr-data       94 Mar 15  2019 Tonsil 1_C26.png -> /uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C26.png

Updating the content of the pattern file

[sbesson@prod97-omeroreadwrite 15-27-40.023]$ cd /uod/idr/metadata/idr0054-segura-tonsilhyperion/
[sbesson@prod97-omeroreadwrite idr0054-segura-tonsilhyperion]$ git diff
diff --git a/experimentA/patterns/Tonsil 1.pattern b/experimentA/patterns/Tonsil 1.pattern
index 132cfd3..8ecce10 100644
--- a/experimentA/patterns/Tonsil 1.pattern     
+++ b/experimentA/patterns/Tonsil 1.pattern     
@@ -1 +1 @@
-/uod/idr/metadata/idr0054-segura-tonsilhyperion/experimentA/patterns/Tonsil 1/Tonsil 1_C<00-26>.png
+Tonsil 1/Tonsil 1_C<00-26>.png

gives a java.io.FileNotFoundException when trying to instantiate the pixel buffer

2021-04-30 13:55:38,700 ERROR [                ome.io.nio.PixelsService] (.Server-17) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023/Tonsil 1.pattern
java.lang.RuntimeException: java.io.FileNotFoundException: Tonsil 1/Tonsil 1_C00.png (No such file or directory)
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79) ~[omero-romio.jar:5.6.2]
    at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124) ~[omero-romio.jar:5.6.2]
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:889) ~[omero-romio.jar:5.6.2]

...
Caused by: java.io.FileNotFoundException: Tonsil 1/Tonsil 1_C00.png (No such file or directory)
    at java.base/java.io.RandomAccessFile.open0(Native Method) ~[na:na]
    at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345) ~[na:na]
    at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259) ~[na:na]
    at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214) ~[na:na]
    at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:130) ~[ome-common.jar:6.0.4]
    at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151) ~[ome-common.jar:6.0.4]
    at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165) ~[ome-common.jar:6.0.4]
joshmoore commented 3 years ago

Doh. Perhaps it's relative to CWD rather than setId?

sbesson commented 3 years ago 
[sbesson@ome-demoserver ~]$ cd /tmp/
[sbesson@ome-demoserver tmp]$ mkdir -p test/data
[sbesson@ome-demoserver tmp]$  echo "data/test_C<0-1>.fake" > test/test.pattern
[sbesson@ome-demoserver tmp]$ touch test/data/test_C0.fake
[sbesson@ome-demoserver tmp]$  touch test/data/test_C1.fake
[sbesson@ome-demoserver tmp]$ cd
[sbesson@ome-demoserver ~]$ /opt/omero/server/OMERO.server/bin/omero import -f /tmp/test/test.pattern 
2021-04-30 15:13:36,127 273        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.8
2021-04-30 15:13:36,148 294        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.5.1 revision: 6f50e4d52c9d96112635fd8b2dde737f31041cf0 date: 7 July 2020
2021-04-30 15:13:36,190 336        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2021-04-30 15:13:36,636 782        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2021-04-30 15:13:36,767 913        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 127ms. (132ms total) [0 unknowns]
#======================================
# Group: /tmp/test/test.pattern SPW: false Reader: loci.formats.in.FilePatternReader
/tmp/test/test.pattern
/fc/homes/sbesson/data/test_C0.fake
/fc/homes/sbesson/data/test_C1.fake

Indeed it looks like the assumption is that the presence of file separator means either absolute or relative to current directory and not the directory of pattern file

sbesson commented 4 months ago

The upgrade is completed with the release of prod122. The pattern based datasets using absolute paths as in https://github.com/IDR/deployment/issues/327#issuecomment-830113542 have been converted to OME-Zarr