search

ome / omero-server

Gradle project containing main server logic for OMERO
https://www.openmicroscopy.org/omero
GNU General Public License v2.0
5 stars 14 forks source link

Bug: min_threads / pixeldata.threads mismatch leads to hang #125

Open joshmoore opened 3 years ago

joshmoore commented 3 years ago

see: https://forum.image.sc/t/pixeldata-threads-and-pyramid-generation-issues/49794

Summary of results: In general, it seems that omero.sessions.timeout does matter, but only when omero.threads.min_threads does not exceed omero.pixeldata.threads, and only when both files are imported in a single import.

chris-allan commented 1 year ago

With the research that is going into #154, specifically looking at the jstack output from @JulianHn on the image.sc thread, I believe at least part of the issue here is a bug with ome.services.pixeldata.PixelDataThread#doRun() when handling exceptional execution conditions. The current code looks like this:

https://github.com/ome/omero-server/blob/5960bc983dd59f165efaa5742126bc340d3881a3/src/main/java/ome/services/pixeldata/PixelDataThread.java#L233-L269

You will notice that the while loop starting on line 256 has a completion condition count > 0 and in the line above the count is set to the number of event logs that were submitted for processing. If one of those event log processing tasks completes exceptionally calling future.get() will throw an ExecutionException ^1 which will then be caught and handled by onExecutionException(). Unfortunately, when this happens count is not decremented and the while loop will proceed indefinitely resulting in no new event logs being processed.

I think we can fix this easily enough while dealing with #154.

Edit: If an InterruptedException is thrown somewhere and caught it will not cause the count to be decremented.

/cc @jburel, @erickmartins, @perlman