Problem with running a CellProfiler Pipeline

[dietzc]

Hi @LeeKamentsky,

a user in our Forum noticed a bug in the CellProfiler integration and I was able to reproduce the bug locally (latest master of CellProfiler and knip-cellprofiler, see https://tech.knime.org/forum/knime-image-processing/problem-with-cellprofiler-integration for example data). The output in my console is as follows:

WARN     CellProfilerInstance            Running module # 22: RelateObjects
WARN     CellProfilerInstance            Running module # 23: RescaleIntensity
WARN     CellProfilerInstance            Running module # 24: OverlayOutlines
WARN     CellProfilerInstance            Running module # 25: OverlayOutlines
WARN     CellProfilerInstance            Finished group
WARN     CellProfilerInstance            
WARN     CellProfilerInstance            Traceback (most recent call last):
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/knime_bridge.py", line 143, in run
WARN     CellProfilerInstance                session_id, message_type, msg)
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/knime_bridge.py", line 428, in run_group_request
WARN     CellProfilerInstance                image_numbers]
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/measurements.py", line 369, in __getitem__
WARN     CellProfilerInstance                return self.get_measurement(*key)
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/measurements.py", line 959, in get_measurement
WARN     CellProfilerInstance                result = self.hdf5_dict[EXPERIMENT, feature_name, 0]
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/utilities/hdf5_dict.py", line 373, in __getitem__
WARN     CellProfilerInstance                result = self[object_name, feature_name, [num_idx]]
WARN     CellProfilerInstance              File "/home/dietzc/devel/git-repositories/CellProfiler/cellprofiler/utilities/hdf5_dict.py", line 377, in __getitem__
WARN     CellProfilerInstance                assert feature_exists
WARN     CellProfilerInstance            AssertionError
ERROR    CellProfiler Pipeline Executor  Execute failed: ("RuntimeException"): 

The thrown exception on java side:

java.lang.RuntimeException: 
    at org.knime.knip.cellprofiler.CellProfilerInstance$2.getCells(CellProfilerInstance.java:316)
    at org.knime.core.data.container.RearrangeColumnsTable.calcNewCellsForRow(RearrangeColumnsTable.java:503)
    at org.knime.core.data.container.RearrangeColumnsTable.calcNewColsSynchronously(RearrangeColumnsTable.java:424)
    at org.knime.core.data.container.RearrangeColumnsTable.create(RearrangeColumnsTable.java:342)
    at org.knime.core.node.ExecutionContext.createColumnRearrangeTable(ExecutionContext.java:369)
    at org.knime.knip.cellprofiler.CellProfilerInstance.execute(CellProfilerInstance.java:167)
    at org.knime.knip.cellprofiler.nodes.pipelineexecutor.PipelineExecutorNodeModel.execute(PipelineExecutorNodeModel.java:55)
    at org.knime.core.node.NodeModel.execute(NodeModel.java:706)
    at org.knime.core.node.NodeModel.executeModel(NodeModel.java:555)
    at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1131)
    at org.knime.core.node.Node.execute(Node.java:927)
    at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
    at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
    at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
    at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
    at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
    at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:125)
    at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:248)
Caused by: org.cellprofiler.knimebridge.CellProfilerException: 
    at org.cellprofiler.knimebridge.message.AbstractReply.recv(AbstractReply.java:56)
    at org.cellprofiler.knimebridge.message.RunGroupReq.run(RunGroupReq.java:56)
    at org.cellprofiler.knimebridge.KnimeBridgeImpl.runGroup(KnimeBridgeImpl.java:166)
    at org.knime.knip.cellprofiler.CellProfilerInstance$2.createCells(CellProfilerInstance.java:386)
    at org.knime.knip.cellprofiler.CellProfilerInstance$2.getCells(CellProfilerInstance.java:312)
    ... 20 more

Do you have any idea what is going on? It seems to me, that the AssertionError on the CellProfiler side somehow causes these problems.

Thanks in advance,

Christian

[LeeKamentsky]

These sorts of problems crop up when a pipeline doesn't generate a measurement that was promised. There are a lot of reasons this can happen. Could you post the pipeline or send it to me? In any case, it's a bug in the knime bridge on the CellProfiler side and I need to write a test case and then fix it.

[dietzc]

test-flow + data can be found at https://tech.knime.org/forum/knime-image-processing/problem-with-cellprofiler-integration. Does this help?

Thanks for the support!!

Christian

[LeeKamentsky]

https://github.com/CellProfiler/CellProfiler/commit/235a5272280724bd4c1fa081a4338e9be234309f and https://github.com/CellProfiler/CellProfiler/commit/13a6470ec5f87be3ee222cdae74de087c18f8b62 fix this. One of the modules was generating an experiment-wide measurement which would not generated by running the pipeline via the bridge. The final fix figures out that this measurement should be excluded, whereas other experiment measurments (like the pipeline that was used and the GIT hash of the CellProfiler source) are reported over the bridge. The user should pick up the trunk build of CellProfiler after it's built this commit. Thanks for reporting.

[dietzc]

amazing. Thanks @LeeKamentsky!