search

BodenmillerGroup / steinbock

A toolkit for processing multiplexed tissue images
https://bodenmillergroup.github.io/steinbock
MIT License
49 stars 14 forks source link

Error when running `steinbock classify ilastik run` #195

Open nilseling opened 1 year ago

nilseling commented 1 year ago

I re-ran the bash script shown here: 

#!/usr/bin/env bash
BASEDIR=$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)
cd "${BASEDIR}"

# raw data collection
mkdir raw
wget https://zenodo.org/record/7997296/files/IMCWorkflow.ilp
wget https://zenodo.org/record/7997296/files/analysis.zip
unzip analysis.zip
rm analysis.zip
rm -r analysis/cpinp
rm -r analysis/cpout
rm -r analysis/histocat
rm -r analysis/ilastik
rm -r analysis/ometiff
cd raw
wget https://zenodo.org/record/7575859/files/panel.csv
wget https://zenodo.org/record/7575859/files/Patient1.zip
wget https://zenodo.org/record/7575859/files/Patient2.zip
wget https://zenodo.org/record/7575859/files/Patient3.zip
wget https://zenodo.org/record/7575859/files/Patient4.zip
cd ${BASEDIR}

# steinbock alias setup
shopt -s expand_aliases
alias steinbock="docker run -v ${BASEDIR}:/data -u $(id -u):$(id -g) ghcr.io/bodenmillergroup/steinbock:0.16.0"

# raw data preprocessing
steinbock preprocess imc panel --namecol Clean_Target
steinbock preprocess imc images --hpf 50

# random forest-based segmentation using Ilastik/CellProfiler
steinbock classify ilastik prepare --cropsize 500 --seed 123
rm pixel_classifier.ilp && mv IMCWorkflow.ilp pixel_classifier.ilp
rm -r ilastik_crops && mv analysis/crops ilastik_crops
steinbock classify ilastik fix --no-backup
steinbock classify ilastik run
steinbock segment cellprofiler prepare
steinbock segment cellprofiler run -o masks_ilastik

In the past (using an older version of the ImcSegmentationPipeline output) worked fine but now I'm getting the following error:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/steinbock/steinbock/__main__.py", line 17, in <module>
    steinbock_cmd_group(prog_name="steinbock")
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/steinbock-venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/steinbock/steinbock/_cli/utils.py", line 34, in wrapper
    return func(*args, **kwargs)
  File "/app/steinbock/steinbock/classification/_cli/ilastik.py", line 336, in fix_cmd
    ilastik.fix_ilastik_project_file_inplace(
  File "/app/steinbock/steinbock/classification/ilastik/_ilastik.py", line 390, in fix_ilastik_project_file_inplace
    _fix_pixel_classification_group_inplace(
  File "/app/steinbock/steinbock/classification/ilastik/_ilastik.py", line 438, in _fix_pixel_classification_group_inplace
    block = np.transpose(block, axes=transpose_axes)
  File "<__array_function__ internals>", line 180, in transpose
  File "/opt/steinbock-venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 660, in transpose
    return _wrapfunc(a, 'transpose', axes)
  File "/opt/steinbock-venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
ValueError: axes don't match array
/opt/ilastik/ilastik-meta/lazyflow/lazyflow/classifiers/__init__.py:20: UserWarning: init: Could not import tiktorch classifier
  warnings.warn("init: Could not import tiktorch classifier")
INFO ilastik_main: Starting ilastik from "/opt/ilastik".
Starting ilastik from "/opt/ilastik".
WARNING 2023-06-12 13:35:28,748 opConservationTracking 31 139775893624640 Could not find any ILP solver
WARNING 2023-06-12 13:35:28,756 opStructuredTracking 31 139775893624640 Could not find any ILP solver
WARNING 2023-06-12 13:35:28,758 structuredTrackingWorkflow 31 139775893624640 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO ilastik.shell.projectManager: Opening Project: pixel_classifier.ilp
WARNING dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING opSimpleBlockedArrayCache.py(43): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING opSlicedBlockedArrayCache.py(141): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
ERROR 2023-06-12 13:35:31,985 log_exception 31 139775893624640 Traceback (most recent call last):
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 460, in _loadProject
    serializer.deserializeFromHdf5(self.currentProjectFile, projectFilePath, self._headless)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 1151, in deserializeFromHdf5
    ss.deserialize(topGroup)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/pixelClassification/pixelClassificationSerializer.py", line 100, in deserialize
    super().deserialize(group)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 264, in deserialize
    self._deserialize(group[self.name], self.inslot)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/utility/timer.py", line 153, in wrapper
    return func(*args, **kwargs)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 622, in _deserialize
    self.inslot[index][slicing] = blockArray
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 988, in __setitem__
    self.operator.setInSlot(self.top_level_slot, self.subindex, roi, value)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/operators/opCompressedUserLabelArray.py", line 394, in setInSlot
    self._setInSlotInput(slot, subindex, roi, new_pixels)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/operators/opCompressedUserLabelArray.py", line 416, in _setInSlotInput
    block_rois = getIntersectingRois(self.Output.meta.shape, self._blockshape, (roi.start, roi.stop))
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 639, in getIntersectingRois
    block_rois = list(map(partial(getBlockBounds, dataset_shape, blockshape), block_starts))
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 667, in getBlockBounds
    block_bounds = getIntersection(block_bounds, entire_dataset_roi)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 402, in getIntersection
    assert ((stop - start) > 0).all(), "Rois do not intersect: {} and {}".format(roiA, roiB)
AssertionError: Rois do not intersect: (array([0, 0, 2]), array([253, 252,   3])) and ([0, 0, 0], [500, 500, 1])

ERROR 2023-06-12 13:35:31,985 log_exception 31 139775893624640 Project could not be loaded due to the exception shown above.
Aborting Project Open Action
ERROR 2023-06-12 13:35:31,986 excepthooks 31 139775893624640 Unhandled exception in thread: 'MainThread'
ERROR 2023-06-12 13:35:31,988 excepthooks 31 139775893624640 Traceback (most recent call last):
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik.py", line 134, in <module>
    main()
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik.py", line 128, in main
    hShell = ilastik_main.main(parsed_args, workflow_cmdline_args)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik_main.py", line 170, in main
    f(shell)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik_main.py", line 347, in loadProject
    shell.openProjectFile(path, parsed_args.readonly)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/shell/headless/headlessShell.py", line 125, in openProjectFile
    self.projectManager._loadProject(hdf5File, projectFilePath, readOnly)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/utility/timer.py", line 153, in wrapper
    return func(*args, **kwargs)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 460, in _loadProject
    serializer.deserializeFromHdf5(self.currentProjectFile, projectFilePath, self._headless)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 1151, in deserializeFromHdf5
    ss.deserialize(topGroup)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/pixelClassification/pixelClassificationSerializer.py", line 100, in deserialize
    super().deserialize(group)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 264, in deserialize
    self._deserialize(group[self.name], self.inslot)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/utility/timer.py", line 153, in wrapper
    return func(*args, **kwargs)
  File "/opt/ilastik/ilastik-meta/ilastik/ilastik/applets/base/appletSerializer.py", line 622, in _deserialize
    self.inslot[index][slicing] = blockArray
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 992, in __setitem__
    p[key] = value
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/slot.py", line 988, in __setitem__
    self.operator.setInSlot(self.top_level_slot, self.subindex, roi, value)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/operators/opCompressedUserLabelArray.py", line 394, in setInSlot
    self._setInSlotInput(slot, subindex, roi, new_pixels)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/operators/opCompressedUserLabelArray.py", line 416, in _setInSlotInput
    block_rois = getIntersectingRois(self.Output.meta.shape, self._blockshape, (roi.start, roi.stop))
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 639, in getIntersectingRois
    block_rois = list(map(partial(getBlockBounds, dataset_shape, blockshape), block_starts))
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 667, in getBlockBounds
    block_bounds = getIntersection(block_bounds, entire_dataset_roi)
  File "/opt/ilastik/ilastik-meta/lazyflow/lazyflow/roi.py", line 402, in getIntersection
    assert ((stop - start) > 0).all(), "Rois do not intersect: {} and {}".format(roiA, roiB)
AssertionError: Rois do not intersect: (array([0, 0, 2]), array([253, 252,   3])) and ([0, 0, 0], [500, 500, 1])

I'm not quite sure where this is coming from. What I changed was the version of the ImcSegmentationPipeline that I used for generating the ilastik crops and I upgraded from Ilastik v1.3.3 to v1.4.0. A similar issue was reported in #188 so it would be good to investigate where this is coming from.

nilseling commented 1 year ago

This can be addressed together with #191

Milad4849 commented 1 year ago

Steinbock docker container is fixed to ilastik-1.3.3, it appears that classifiers from ilastik 14.0 and above are not compatible with the older versions. That is the cause of this and #188.

Milad4849 commented 1 year ago

Given above is this actually a bug? Shall we update ilastik with the next release? @jwindhager

nilseling commented 1 year ago

Did you check if updating to 1.4.0 fixes the issue?