sbesson
commented
2 years ago Is the processor the only faulty service? Could you paste the output of `omero admin diagnostics?
Closed pwalczysko closed 2 years ago
sbesson
commented
2 years ago Is the processor the only faulty service? Could you paste the output of `omero admin diagnostics?
pwalczysko
commented
2 years ago Is the processor the only faulty service? Could you paste the output of `omero admin diagnostics?
Afaik, Tables and Indexer services are working fine (viewed an OMERO.table on an image and created a new KVP and successfully searched for it). I suppose if Blitz would be faulty, I could not do very much at all ? (now I am logged in to fully functioning OMERO.web)
(venv3) [pwalczysko@ome-training-4 ~]$ omero admin diagnostics
/opt/omero/server/OMERO.server/etc/grid/config.xml read-only
================================================================================
OMERO Diagnostics (admin) 5.10.1
================================================================================
Commands: java -version 1.8.0 (/bin/java)
Commands: python -V 2.7.5 (/bin/python -- 2 others)
Commands: icegridnode --version 3.6.5 (/bin/icegridnode)
Commands: icegridadmin --version 3.6.5 (/bin/icegridadmin)
Commands: psql --version 10.19 (/bin/psql)
Commands: openssl version 1.0.2 (/bin/openssl)
Server: icegridnode running
Server: Blitz-0 active (pid = 11901, enabled)
Server: Indexer-0 active (pid = 11912, enabled)
Server: OMERO.Glacier2 active (pid = 11913, enabled)
Server: Processor-0 active (pid = 11915, enabled)
Server: Tables-0 active (pid = 11924, enabled)
Log dir: /opt/omero/server/OMERO.server/var/log exists
Log files: Blitz-0.log 442.1 MB errors=51 warnings=1001
Log files: Blitz-0.log.1 524.3 MB errors=0 warnings=58
Log files: DropBox.log 2.8 KB errors=0 warnings=3
Log files: FileServer.log 555 B errors=0 warnings=2
Log files: Indexer-0.log 1.2 MB errors=0 warnings=42
Log files: MonitorServer.log 16.6 KB errors=0 warnings=2
Log files: PixelData-0.log 404.6 KB errors=0 warnings=2
Log files: Processor-0.log 3.9 MB errors=0 warnings=99
Log files: Tables-0.log 18.5 KB errors=0 warnings=3
Log files: TestDropBox.log n/a
Log files: master.err 1.6 KB errors=1 warnings=1
Log files: master.out empty
Log files: Total size 971.92 MB
Environment:OMERO_HOME=(unset)
Environment:OMERODIR=/opt/omero/server/OMERO.server
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/bin:/opt/omero/server/venv3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/homes/pwalczysko/.local/bin:/homes/pwalczysko/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)
OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? False
OMERO temp dir:'/homes/pwalczysko/omero/tmp' Exists? True Is writable? True (Size: 0)
JVM settings: Blitz-${index} -Xmx16783m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx6713m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx6713m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx3356m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
Jar: lib/server/formats-api.jar Bio-Formats API 6.9.1 20 April 2022 ceb7dcecff23ce0c9be1e82244b6ee78beffab0a
Jar: lib/server/formats-bsd.jar BSD Bio-Formats readers and writers 6.9.1 20 April 2022 ceb7dcecff23ce0c9be1e82244b6ee78beffab0a
Jar: lib/server/formats-gpl.jar Bio-Formats library 6.9.1 20 April 2022 ceb7dcecff23ce0c9be1e82244b6ee78beffab0a
Jar: lib/server/ome-codecs.jar OME Codecs 0.3.1 23 June 2021 3a52a459c9e10c6a67b522c56d3b0a727c904da6
Jar: lib/server/ome-common.jar OME Common Java 6.0.7 18 June 2021 739f1483ffdbd07a74b04ef12752d6e7737eb056
Jar: lib/server/ome-jai.jar OME JAI 0.1.0 28 June 2017 96ed34bf59f5ba3b51e3d60e342f30962a46c292
Jar: lib/server/ome-mdbtools.jar MDB Tools (Java port) 5.3.2 7 August 2018 68ffca2a95750cbb96f0d11785851aa4a554c0a4
Jar: lib/server/ome-poi.jar OME POI 5.3.4 23 June 2021 5c798fe02168cec417a124f8baca757eb292d2a6
Jar: lib/server/ome-xml.jar OME XML library 6.2.3 23 June 2021 c3f5be6a8e134db2480804551a2802e8fb2ab535
Jar: lib/server/omero-blitz.jar jar 5.5.10
Jar: lib/server/omero-common.jar jar 5.5.9
Jar: lib/server/omero-gateway.jar jar 5.6.9
Jar: lib/server/omero-model.jar jar 5.6.5
Jar: lib/server/omero-renderer.jar jar 5.5.9
Jar: lib/server/omero-romio.jar jar 5.6.4
Jar: lib/server/omero-server.jar jar 5.6.3
Something called BlitzAdapter is possibly not working @sbesson ? Please see below cc @joshmoore
[pwalczysko@ome-training-4 log]$ cat master.err
-! 21/06/22 13:47:07:556 warning: Blitz-0-Ice.ThreadPool.Server-3846: dispatch exception:
identity: session-c80a9033-c2be-437a-b32a-cac379d0e0fb/23b99750-f304-47ae-8cf7-d9bfb67ca32a
facet:
operation: destroy
remote host: 172.17.0.1 remote port: 50926
Ice.ObjectAdapterDeactivatedException
name = "BlitzAdapter"
at Ice.ObjectAdapterI.checkForDeactivation(ObjectAdapterI.java:1170)
at Ice.ObjectAdapterI.findFacet(ObjectAdapterI.java:498)
at Ice.ObjectAdapterI.find(ObjectAdapterI.java:491)
at omero.cmd.SessionI.unregisterServant(SessionI.java:658)
at omero.cmd.SessionI.cleanServants(SessionI.java:461)
at omero.cmd.SessionI.cleanServants(SessionI.java:391)
at omero.cmd.SessionI.doDestroy(SessionI.java:381)
at omero.cmd.SessionI.destroy(SessionI.java:309)
at omero.api._ServiceFactoryTie.destroy(_ServiceFactoryTie.java:65)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:94)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1263)
at IceInternal.Incoming.invoke(Incoming.java:221)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
at Ice.ConnectionI.message(ConnectionI.java:1056)
at IceInternal.ThreadPool.run(ThreadPool.java:395)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
at java.lang.Thread.run(Thread.java:748)
From Processor-0.log
2022-06-21 13:47:07,587 WARNI [ omero.util.ServerContext] (MainThread) Connection failure: exception ::Ice::NoEndpointException
{
proxy = session-efe2093a-3310-431e-9a0d-e31445de5680/cfe5ca59-c04e-4ad5-bc65-7492783bdc1e -t -e 1.1 @ BlitzAdapters
}
joshmoore
commented
2 years ago ObjectAdapterDeactivatedException occurs during shutdown. Does the timestamp match a restart?
pwalczysko
commented
2 years ago ObjectAdapterDeactivatedException occurs during shutdown. Does the timestamp match a restart?
Thanks. Yes, it matches the time when I ran the playbook. So this is a red herring, i.e. not the cause of the problem ?
joshmoore
commented
2 years ago So this is a red herring, i.e. not the cause of the problem ?
I believe so.
pwalczysko
commented
2 years ago Test:
Not sure where to go from here.
sbesson
commented
2 years ago After some trial and error, it looks like Storm is a requirement for the Processor server to be functional. Below is the config that worked for me in the end.
omero.server.nodedescriptors master:Blitz-0,Indexer-0,Processor-0,Storm,Tables-0I reverted my change and restarted the server, do you want to give it a try?
pwalczysko
commented
2 years ago Thank you, will rerun tomorrow.
pwalczysko
commented
2 years ago Applied the changes in the last commit to ome-training-4. Scripts, search of newly created KVPs, expanding of Tables harmonica and viewing tables using "eye" icon and general webclient work fine.
The Pixel data tests:
I think this is good, @sbesson ?
pwalczysko
commented
2 years ago Deployed successfully on all 4 training servers. All seems to work. Thank you. Merging.
See https://github.com/ome/prod-playbooks/issues/353 for background.
@sbesson I am afraid the suggested change as implemented in this PR and tested on ome-training-4 is causing "No processor available" when trying to run a script (Batch image export).