Segmentations have "holes" in them

[paul-reiners]

Some of our segmentations created by nnU-Net models have "holes" in them. That is, they contain voxels in the interior of the brain which are labeled as 0 or background.

Initial segmentations produced by nnUNet don't have holes until the file is modified during the "predicting" and/or "postprocessing" stages.

I noticed something very strange regarding the gaping holes we suddenly started to observe in the segmentations with the original model. I was running one of the new models (552) and noticed that the segmentation files look good once the "preprocessing stage" is complete, ie the following from the logs:

starting preprocessing generator
starting prediction...
preprocessing /home/feczk001/shared/projects/segpipeline_testing/nnUNet_new_model_testing/outputs_552/sub-xxx_ses-10mo_xfmspreBIBSnet_final.nii.gz
using preprocessor GenericPreprocessor
before crop: (2, 182, 218, 182) after crop: (2, 182, 218, 166) spacing: [1. 1. 1.] 

no separate z, order 3
no separate z, order 1
before: {'spacing': array([1., 1., 1.]), 'spacing_transposed': array([1., 1., 1.]), 'data.shape (data is transposed)': (2, 182, 218, 166)} 
after:  {'spacing': array([0.799999, 0.799999, 0.799999]), 'data.shape (data is resampled)': (2, 228, 273, 208)} 

(2, 228, 273, 208)
This worker has ended successfully, no errors to report

but then after the "predicting" and "postprocessing" stages, the holes appear. see before and after examples for 2 sessions. The "before" file was one that I copied to a separate folder before nnUNet completed:

Zoom before_ses4.png

Zoom after_ses4.png

Zoom before_ses10.png

Zoom after_ses10.png

I know nnUNet is kind of a black box, but I'm curious to know what the preprocessing, predict, and postprocessing steps even mean in the context of running inference and I wonder if it might be worth stepping through the code to idenitfy at which point the holes are introduced. The original model seemed be give us more detailed segmentations than the new models trained with a larger number of SynthSeg images, we just couldn't use it because of the holes issue

Here are the remaining logs from the "predicting" and "postprocessing" stages, which run at the end once all initial file outputs are produced:

force_separate_z: None interpolation order: 1
separate z: False lowres axis None
no separate z, order 1
force_separate_z: None interpolation order: 1
separate z: False lowres axis None
no separate z, order 1
force_separate_z: None interpolation order: 1
separate z: False lowres axis None
no separate z, order 1
predicting /home/feczk001/shared/projects/segpipeline_testing/nnUNet_new_model_testing/outputs_552/sub-xxx_ses-10mo_xfmspreBIBSnet_final.nii.gz
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
computing Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
This output is too large for python process-process communication. Saving output temporarily to disk
predicting /home/feczk001/shared/projects/segpipeline_testing/nnUNet_new_model_testing/outputs_552/sub-xxx_ses-4mo_xfmspreBIBSnet_final.nii.gz
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 210)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 38, 76, 114]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 210)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 38, 76, 114]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 210)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 38, 76, 114]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 210)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 38, 76, 114]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 210)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 38, 76, 114]]
number of tiles: 64
using precomputed Gaussian
prediction done
This output is too large for python process-process communication. Saving output temporarily to disk
predicting /home/feczk001/shared/projects/segpipeline_testing/nnUNet_new_model_testing/outputs_552/sub-xxx_ses-1mo_xfmspreBIBSnet_final.nii.gz
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
debug: mirroring True mirror_axes (0, 1, 2)
step_size: 0.5
do mirror: True
data shape: (2, 228, 273, 208)
patch size: [112 128  96]
steps (x, y, and z): [[0, 39, 77, 116], [0, 48, 97, 145], [0, 37, 75, 112]]
number of tiles: 64
using precomputed Gaussian
prediction done
This output is too large for python process-process communication. Saving output temporarily to disk
inference done. Now waiting for the segmentation export to finish...
postprocessing...

[ericfeczko]

@paul-reiners you need to add the PNGs so folks can see them

[Karol-G]

Hey @paul-reiners,

I am part of the nnU-Net support team. If I understand you correctly, you mean to say that the ground truth does not have holes in the interior of the brain, but the predictions inferred by nnU-Net have?

Initial segmentations produced by nnUNet don't have holes until the file is modified during the "predicting" and/or "postprocessing" stages.

I am not sure if I understand you correctly here. What do you mean by initial segmentation?

Is it possible for you to upload a prediction with these holes and the corresponding ground truth so I can understand the problem better? If you want you can send them to karol.gotkowski@dkfz.de.

Best, Karol

[ericfeczko]

Hey @Karol-G ,

To clarify, the segmentations looks great out of nnunet_predict, minus an issue with chirality -- left and right labels sometimes get swapped -- but the postprocessing stage produces holes in the segmentation.

I think we figured out a resolution to our problem. The postprocessing involves eliminating discontiguous elements of the segmentation, which ordinarily reflects false positives (e.g. in tumor detection).

For our segmentations, the predicted segmentation can have a chriality issue, where the left and right labels are swapped. This is trivial for us to correct. However, if it is not corrected before postprocessing, postprocessing recognizes the swapped labels as discontiguous and removes them.

[Karol-G]

Ah, I see. That really seems like a unique issue that requires a tailored solution like the one you proposed. You can also try to simply disable the postprocessing. Even under normal circumstances, the postprocessing improves the predictions only minimally.

[tjhendrickson]

Hi @Karol-G ,

Wanted to revitalize this thread a bit. While we solved the "hole" issue, we continue to deal with the chirality -- where left and right hemisphere labels get swapped.

Are there reasons endemic to the training data itself that could be causing this that you can think of?

Happy to provide provide screenshots of anything you need by the way to aid you.

Best,

-Tim

[Karol-G]

Hey @tjhendrickson,

Sorry for the late reply! I had some deadlines last week and had to prioritize. Regarding the label swapping, it will probably be difficult finding a cause for that as it sounds like quite a unique problem. Maybe this is related to the patch-based training and inference of nnU-Net and the model does not have enough context within a patch to accurately determine if it is seeing the left or right hemisphere. Could you provide your nnUNetPlans.json from the nnUNet_preprocessed folder so that I can have a look? Maybe increasing the patch size would mitigate the problem as it gives the model more spatial context.

Best regards, Karol