Runtime error when segmenting image

[mathieuboudreau]

Here's the command & output:

(nnunet) mathieuboudreau@Mathieus-MacBook-Pro nn-axondeepseg % python nn_axondeepseg.py --seg-type UM --path-out output --path-dataset input --path-model models/model_seg_unmyelinated_tem
perform_everything_on_gpu=True is only supported for cuda devices! Setting this to False
2023-10-26 12:51:57.348 | INFO     | __main__:main:91 - Running inference on device: cpu
use_folds is None, attempting to auto detect available folds
found the following folds: [3]
2023-10-26 12:51:59.552 | INFO     | __main__:main:94 - Model loaded successfully.
2023-10-26 12:51:59.552 | INFO     | __main__:main:101 - Creating temporary input directory.
There are 1 cases in the source folder
I am process 0 out of 1 (max process ID is 0, we start counting with 0!)
There are 1 cases that I would like to predict
Process SpawnProcess-11:
Traceback (most recent call last):
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/inference/data_iterators.py", line 57, in preprocess_fromfiles_save_to_queue
    raise e
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/inference/data_iterators.py", line 31, in preprocess_fromfiles_save_to_queue
    data, seg, data_properties = preprocessor.run_case(list_of_lists[idx],
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 139, in run_case
    data, seg = self.run_case_npy(data, seg, data_properties, plans_manager, configuration_manager,
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 78, in run_case_npy
    data = self._normalize(data, seg, configuration_manager,
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 183, in _normalize
    scheme = configuration_manager.normalization_schemes[c]
IndexError: list index out of range
Traceback (most recent call last):
  File "/Users/mathieuboudreau/neuropoly/github/nn-axondeepseg/nn_axondeepseg.py", line 123, in <module>
    main()
  File "/Users/mathieuboudreau/neuropoly/github/nn-axondeepseg/nn_axondeepseg.py", line 111, in main
    predictor.predict_from_files(str(tmp_dir), args.path_out)
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 249, in predict_from_files
    return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 342, in predict_from_data_iterator
    for preprocessed in data_iterator:
  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/inference/data_iterators.py", line 109, in preprocessing_iterator_fromfiles
    raise RuntimeError('Background workers died. Look for the error message further up! If there is '
RuntimeError: Background workers died. Look for the error message further up! If there is none then your RAM was full and the worker was killed by the OS. Use fewer workers or get more RAM in that case!

MacBookPro 2021, CPU (no CUDA), Python 3.9 pip env:

(nnunet) mathieuboudreau@Mathieus-MacBook-Pro nn-axondeepseg % pip freeze
acvl-utils==0.2
batchgenerators==0.25
certifi==2023.7.22
charset-normalizer==3.3.1
connected-components-3d==3.12.3
contourpy==1.1.1
cycler==0.12.1
dicom2nifti==2.4.8
dynamic-network-architectures==0.2
filelock==3.12.4
fonttools==4.43.1
fsspec==2023.10.0
future==0.18.3
graphviz==0.20.1
idna==3.4
imagecodecs==2023.9.18
imageio==2.31.6
importlib-resources==6.1.0
Jinja2==3.1.2
joblib==1.3.2
kiwisolver==1.4.5
lazy_loader==0.3
linecache2==1.0.0
loguru==0.7.2
MarkupSafe==2.1.3
matplotlib==3.8.0
mpmath==1.3.0
networkx==3.2
nibabel==5.1.0
nnunetv2==2.2
numpy==1.26.1
opencv-python==4.8.1.78
packaging==23.2
pandas==2.1.1
Pillow==10.0.1
pydicom==2.4.3
pyparsing==3.1.1
python-dateutil==2.8.2
python-gdcm==3.0.22
pytz==2023.3.post1
PyYAML==6.0.1
requests==2.31.0
scikit-image==0.22.0
scikit-learn==1.3.2
scipy==1.11.3
seaborn==0.13.0
SimpleITK==2.3.0
six==1.16.0
sympy==1.12
threadpoolctl==3.2.0
tifffile==2023.9.26
torch==2.1.0
tqdm==4.66.1
traceback2==1.4.0
typing_extensions==4.8.0
tzdata==2023.3
unittest2==1.1.0
urllib3==2.0.7
yacs==0.1.8
zipp==3.17.0

[hermancollin]

Hm I expect the problem comes from the fact that the input image is not formatted like the model expects. This was something I was worried about because we should tell the user why this is failing.

  File "/Users/mathieuboudreau/opt/anaconda3/envs/nnunet/lib/python3.9/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 183, in _normalize
    scheme = configuration_manager.normalization_schemes[c]
IndexError: list index out of range

For the unmyelinated TEM model, when we look at the dataset.json file, we see the following:

{
    "channel_names": {
        "0": "rescale_to_0_1"
    },
    [...]
}

So for this one, we expect grayscale single channel images. However, if we look at the other model available, we have this:

{
    "channel_names": {
        "0": "R",
        "1": "G",
        "2": "B"
    },
    [...]
}

This is used for the image preprocessing. rescale_to_0_1 will rescale the histogram, while 'R', 'G' and 'B' are other normalization schemes (see https://github.com/MIC-DKFZ/nnUNet/blob/master/nnunetv2/dataset_conversion/Dataset120_RoadSegmentation.py). Would you mind posting the image you are trying to segment? I would bet it's in RGB format or maybe it has an alpha channel.

[mathieuboudreau]

Ahh that makes sense ! The image I was trying it with was just our default SEM model test image

[mathieuboudreau]

This one

[hermancollin]

Ah yes this image indeed has 3 channels. I think this should be caught by the script and send a warning to the user instead of crashing with this unhelpful error message.

[mathieuboudreau]

Ah yes this image indeed has 3 channels. I think this should be caught by the script and send a warning to the user instead of crashing with this unhelpful error message.

Maybe this is an example of why #1 might be useful, because we already have the tools to do this conversion (RGB -> Gray) there

[hermancollin]

Ah yes this image indeed has 3 channels. I think this should be caught by the script and send a warning to the user instead of crashing with this unhelpful error message.

Maybe this is an example of why #1 might be useful, because we already have the tools to do this conversion (RGB -> Gray) there

Yeah and the nnUNet pipeline requires creating a temp folder with different filenames anyway so we can for sure convert the color profile at this step.