search

CellProfiler / CellProfiler-plugins

Community-contributed and experimental CellProfiler modules.
http://plugins.cellprofiler.org/
56 stars 67 forks source link

Pipeline frozen on post processing after RunCellpose with Docker #251

Open emiglietta opened 1 month ago

emiglietta commented 1 month ago

Version of Cellprofiler: 4.2.6 (source installation)

After running a pipeline (on Analysis mode) containing two RunCellpose modules running with Docker, the pipeline gets stuck on post processing. This has happened to me sometimes even when the pipeline runs with no errors, but the only way of consistently reproducing this error was to run the pipeline with more workers than CPUs available and induce the RunCellpose module to fail (the easiest way is to decrease the total memory available for Docker) and, when it does, choose continue with analysis, skipping the failing images.

emiglietta commented 1 month ago

I managed to get this error consistently without it failing at any point (except for the freezing during 'post-processing'). When I analyzed just a few image sets (up to 20), it never froze. However, when I tried analyzing more sets (as low as 50), then it consistently froze during post-processing.

bethac07 commented 1 month ago

@emiglietta have you checked how, if at all, running with "automatically continue after failing images" turned on affects the outcome? Might be a potentially diagnostic clue (and a good workaround, if it resolves the issue temporarily)

emiglietta commented 1 month ago

Thanks for the suggestion @bethac07! I tried running the pipeline with the "always skip failing images" option ON, but the result was the same: the pipeline got stuck in "post-processing" after trying to analyze 50 image sets.