search

hammerlab / cytokit

Microscopy Image Cytometry Toolkit
Apache License 2.0
115 stars 18 forks source link

IndexError: too many indices for array #20

Closed mkeays closed 4 years ago

mkeays commented 4 years ago

Hello,

I'm now trying to run Cytokit to do some processing of a set of tiles I generated (using bfconvert) from a single large fluorescence microscopy image with 4 channels, one z-plane.

I have 400 tiles (20x20), with one cycle, one z-plane, 4 channels, x = 2633, y = 1802. I put the tiles into output/processor/tile and then tried to run cytokit processor run_all --config-path=experiment.yaml --data-dir=output --output-dir=output. The pipeline then fails with the following:

2020-01-20 11:49:42,394:INFO:21480:cytokit.exec.pipeline: Starting Pre-processing pipeline for 2 tasks (2 workers) Using TensorFlow backend. Using TensorFlow backend. /lab/repos/cytokit/python/pipeline/cytokit/io.py:137: UserWarning: ImageJ tags do not contain "axes" property (file = /users/keays/cytokit/all_tiles/output/processor/tile/R001_X001_Y011.tif, tags = {'ImageJ': '', 'slices': 1, 'images': 4, 'hyperstack': True, 'frames': 1, 'channels': 4}) warnings.warn('ImageJ tags do not contain "axes" property (file = {}, tags = {})'.format(file, tags)) /lab/repos/cytokit/python/pipeline/cytokit/io.py:137: UserWarning: ImageJ tags do not contain "axes" property (file = /users/keays/cytokit/all_tiles/output/processor/tile/R001_X001_Y001.tif, tags = {'ImageJ': '', 'slices': 1, 'images': 4, 'hyperstack': True, 'frames': 1, 'channels': 4}) warnings.warn('ImageJ tags do not contain "axes" property (file = {}, tags = {})'.format(file, tags)) Exception in thread Thread-3: Traceback (most recent call last): File "/opt/conda/envs/cytokit/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/opt/conda/envs/cytokit/lib/python3.5/threading.py", line 862, in run self._target(*self._args, *self._kwargs) File "/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 169, in load_tiles q.put((tile, region_index, tile_index), block=True, timeout=TIMEOUT) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 157, in exit raise value File "/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 167, in load_tiles tile = op.run(None) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 178, in run res = self._run(args, **kwargs) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/tile_generator.py", line 64, in _run tile = cytokit_io.read_tile(osp.join(self.data_dir, img_path)) File "/lab/repos/cytokit/python/pipeline/cytokit/io.py", line 152, in read_tile res = tif.asarray()[tuple(slices)] IndexError: too many indices for array

Exception in thread Thread-3: Traceback (most recent call last): File "/opt/conda/envs/cytokit/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/opt/conda/envs/cytokit/lib/python3.5/threading.py", line 862, in run self._target(*self._args, *self._kwargs) File "/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 169, in load_tiles q.put((tile, region_index, tile_index), block=True, timeout=TIMEOUT) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 157, in exit raise value File "/lab/repos/cytokit/python/pipeline/cytokit/exec/pipeline.py", line 167, in load_tiles tile = op.run(None) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/op.py", line 178, in run res = self._run(args, **kwargs) File "/lab/repos/cytokit/python/pipeline/cytokit/ops/tile_generator.py", line 64, in _run tile = cytokit_io.read_tile(osp.join(self.data_dir, img_path)) File "/lab/repos/cytokit/python/pipeline/cytokit/io.py", line 152, in read_tile res = tif.asarray()[tuple(slices)] IndexError: too many indices for array

I looked a bit more closely at the TIFFs with tifffile and it does look as though they just have three dimensions, rather than five. I suppose this is just because there is only one single z-plane and "cycle", so there are no dimensions in the array for these. Looking at /lab/repos/cytokit/python/pipeline/cytokit/io.py , if I manually run the initial steps of read_tile and remove lines 149 and 150, then line 152 only tries to select three dimensions from the image array and so "works". I'm not sure what knock-on effects that would have though...

Would you recommend trying to add placeholder dimensions to the TIFFs somehow for these, or is there a better way to handle these images?

mkeays commented 4 years ago

To update on this, I ended up dumping out each channel to a separate TIFF and treating the files as raw data instead of processed tiles, and now the pipeline is running, albeit with a lot of warnings about invalid ImageJ metadata.

eric-czech commented 4 years ago

Hey @mkeays, running the pipeline from tiles assembled elsewhere like in the CODEX spleen example should be possible and it looks like you've already got the correct run_tile_generator: False property in the config in order to hit that code path. There is logic in there that detects 3D tiffs and expands them to 5D, but it relies on imagej metadata tags. The code for that is right around this related warning though the warning itself is pretty vague ("ImageJ tags do not contain "axes" property blah blah ...").

In the future, if you were to somehow load and save the tiles with ImageJ instead it should work. I'm not familiar with bfconvert so I'm not sure how it puts the necessary metadata into the tiffs. If you can work it out though, it would be a pretty easy addition in that read_tile method.