Error during preprocessing

[mkeays]

Hello,

I am trying Cytokit with some CODEX data from a collaborator, and have run into the error below and I'm not sure how to overcome it. Would you have any idea what might be causing this?

Thanks, Maria

2019-11-26 10:52:38,705:INFO:69377:root: Execution arguments and environment saved to "/nfs/cg/hm/cytokit_testing/analysis/dataset2/output/processor/execution/201911261052.json" 2019-11-26 10:52:47,400:INFO:69377:cytokit.exec.pipeline: Starting Pre-processing pipeline for 2 tasks (2 workers) Using TensorFlow backend. Using TensorFlow backend. distributed.worker - WARNING - Compute Failed Function: run_preprocess_task args: ({'op_flags': <cytokit.exec.pipeline.OpFlags object at 0x7fcda3353080>, 'tile_prefetch_capacity': 1, 'output_dir': '/nfs/cg/hm/cytokit_testing/analysis/dataset2/output', 'data_dir': '/nfs/cg/hm/cytokit_testing/analysis/dataset2/data', 'region_indexes': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 'gpu': 2, 'tile_indexes': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])}) kwargs: {} Exception: ValueError('A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 250, 336, 512), (None, 251, 336, 256)]',) Traceback (most recent call last): File "/lab/repos/cytokit/python/pipeline/cytokit/cli/main.py", line 32, in main() File "/lab/repos/cytokit/python/pipeline/cytokit/cli/main.py", line 28, in main fire.Fire(Cytokit) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/fire/core.py", line 127, in Fire component_trace = _Fire(component, args, context, name) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/fire/core.py", line 366, in _Fire component, remaining_args) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/fire/core.py", line 542, in _CallCallable result = fn(varargs, kwargs) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cli/init.py", line 167, in run_all fn({config[op], params}) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cli/processor.py", line 131, in run pipeline.run(pl_config, logging_init_fn=self._logging_init_fn) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 458, in run run_tasks(pl_conf, 'Pre-processing', run_preprocess_task, logging_init_fn) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 421, in run_tasks res = [r.result() for r in res] File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 421, in res = [r.result() for r in res] File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/distributed/client.py", line 227, in result six.reraise(result) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/six.py", line 692, in reraise raise value.with_traceback(tb) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 441, in run_preprocess_task return run_task(task, ops, preprocess_tile) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 355, in run_task with ops: File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 200, in enter v.enter() File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 152, in enter self.initialize() File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/ops/cytometry.py", line 136, in initialize self.cytometer.initialize() File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cytometry/cytometer.py", line 609, in initialize self.model = self._get_model(input_shape) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cytometry/cytometer.py", line 885, in _get_model return unet_model.get_model(3, input_shape) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cytometry/models/unet_v2.py", line 82, in get_model [x, y] = get_model_core(n_class, input_shape, kwargs) File "/nfs/cg/hm/cytokit_testing/lab/repos/cytokit/python/pipeline/cytokit/cytometry/models/unet_v2.py", line 47, in get_model_core y = keras.layers.merge.concatenate([d, c]) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/keras/layers/merge.py", line 649, in concatenate return Concatenate(axis=axis, kwargs)(inputs) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/keras/engine/base_layer.py", line 431, in call self.build(unpack_singleton(input_shapes)) File "/opt/conda/envs/cytokit/lib/python3.5/site-packages/keras/layers/merge.py", line 362, in build 'Got inputs shapes: %s' % (input_shape)) ValueError: A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 250, 336, 512), (None, 251, 336, 256)]

[eric-czech]

Hi @mkeays ,

The issue is that the CellProfiler U-Net requires the height and width of the images to be evenly divisible by 2 raised to the number of layers in the network, which in this case works out to be 2^3 = 8. Are your images actually 251 x 336 (rows x cols) after ignoring overlap? That seems pretty small so perhaps it's a configuration error. If that is correct though, then you can set the "target_shape" property like it is set here. This will resize the image before sending it to the U-Net to avoid that error and based on what I'm seeing, I would suggest setting that to [256, 336] (though I would first make sure you're really expecting them to be that small).

[eric-czech]

For posterity, the issue was also raised at https://github.com/CellProfiler/CellProfiler-plugins/issues/65.

[mkeays]

Hi @eric-czech , thank you very much for explaining this. The tiles are 1007 x 1344 without overlap, and setting the target_shape to the [504, 672] seems to have solved it and the pipeline is now progressing.

[eric-czech]

Another important caveat I have found with that network is that images really need to be in the ballpark of 20x magnification for it to work well. In other words, if your original images were not taken at 40x and then downsampled by about 1/2 such as in that example I linked to, you may very well not get good results. I think what you did is best if you were using a 40x objective but if it was 20x, then I'd change the target_shape to [1024, 1344].

[eric-czech]

One last thought: Assuming that works, you might also consider setting this property in your configuration (for CP Analyst exports): https://github.com/hammerlab/cytokit/blob/master/pub/config/codex-spleen/experiment.yaml#L129

The "Explorer" app with Cytokit is useful for quick QC for properties like that target_shape (i.e. it's probably the fastest way to get a fast sense of whether or not the segmentation worked well) but for going beyond that CP Analyst is definitely the best option if you were looking for a GUI (though it will slow the processing down a lot).

[mkeays]

Hi @eric-czech , thanks for the further explanation. This data is 20x so I will try [1024, 1344] for the target_shape as you suggest. My ultimate goal is to build an automatic pipeline with minimal manual input but I will take a look with the GUI for QC purposes.

[eric-czech]

Sounds good, I'll close this out then.