Inference with trained model produces null results on training and test images

[PatrickGonzalez]

Dear Fabian,

First and foremorst, thank you for all the wonderful work you have done on the nnUNet!

I have the following issue however: I have a segmentation task on 2D rgb images: segmenting one retinal layer in polarization sensitive OCT images. I processed the images according to the Roadmap dataset example so that they are in the right format in the ImagesTr/ImagesTs/labelsTr/labelsTs folder. The prepocessing is then also done as per Roadmap example.

The 2D network runs for 50 epochs and I get very nice results in the validation set.

If I now try to run inference with the trained model on the test images and the training images (which include the validation images) I get null results only.

Any help would be very much appreciated.

Patrick González

[FabianIsensee]

Hey, are you absolutely certain that you processed the images you used for inference the same way as you did when you converted the dataset into nnU-Net format? If that's the case I don't know how I could help you without actually looking at the dataset. Would that be possible? Best, Fabian

[PatrickGonzalez]

Dear Fabian,

Thank you for your swift reply. I understood that nnUNET_predic does the preprocessing so you can run it on the raw data? I included my raw_data folder for this specific task. It contains the training/test images and labels as well as the dataset.json file.

Task106_OptAxis_Sclera.zip t

[FabianIsensee]

Oof man. You found a really obscure bug in nnU-Net. Luckily it doesn't really affect any datasets except yours so it's not too bad haha. Unfortunately however it's rather involved to fix this properly because it would require a lot of changes - rather inconvenient haha.

To work around the bug you need to force nnU-Net to not run cropping during preprocessing. The way you can do that is by simply setting the corners of the images to 1 instead of the original pixel values (0). For example like this:

import SimpleITK as sitk
from batchgenerators.utilities.file_and_folder_operations import *
from nnunet.paths import nnUNet_raw_data
from nnunet.utilities.task_name_id_conversion import convert_id_to_task_name

def load_nifti_add_workaround(image_file: str):
    a = sitk.ReadImage(image_file)
    b = sitk.GetArrayFromImage(a)
    b[0, 0, 0] = 1
    b[0, -1, -1] = 1
    c = sitk.GetImageFromArray(b)
    c.CopyInformation(a)
    sitk.WriteImage(c, image_file)

if __name__ == '__main__':
    imagestr = join(nnUNet_raw_data, convert_id_to_task_name(106), 'imagesTr')
    imagests = join(nnUNet_raw_data, convert_id_to_task_name(106), 'imagesTs')
    for folder in (imagestr, imagests):
        nii_files = subfiles(folder, join=True, suffix='.nii.gz')
        for n in nii_files:
            load_nifti_add_workaround(n)

Note that you will need to apply this to all images that you plan to predict with nnU-Net. They will always need the 1's in the corners!

Once you have this in place the model produces a more normal training. And the predictions also make sense now :-)

(PSO15 prediction)

One last thing: Your dataset looks very strange (see above). It looks like many pixels were set to 0. This strange appearance is actually also what's causing the bug in nnU-Net to appear :upside_down_face: . Could it be that there is a bug in your conversion to nifti? The road segmentation example casts to uint8 if I remember correctly. You should probably remove that or better replace with with a cast to float32 (.astype(np.float32)).

Best, Fabian

[PatrickGonzalez]

Hi Fabian,

wow! I re-trained the network with the workaround, and the inference runs as it should. Thank you for your great help!

Just to give you an idea: the images and segmentations come from a matlab .mat file. I noticed that in every scanmany pixels were nan's. I savesd this scans first as .png files with imageio.imsave after casting them to uint8. These png's where then preprocessed as in the road segmentation dataset. In the roadsegmentation conversion the segmentations are cast to uint32, the images aren't casted to any specific type. I should probably cast them to float32 before saving them as pngs I guess.

Thank you for your help.

Best,

Patrick

[FabianIsensee]

Happy to hear that is works as intended now. I still have the feeling that something is off with your images. When you open the original images in the program provided by the OCT device do they still look like in my screenshot above? Then I search for sclera OCT in google images the results I get look very different from your images:

I think you could be leaving a lot of segmentation performance on the table as long as the images are not the way they should be :-)

I am not sure whether png supports float values. Can't you just export as nifti from matlab? That should be possible

[PatrickGonzalez]

HI Fabian,

Thanks for the feedback. The images the network is trained on carry polarization information of the light reflected back by the various retinal layers. That is why they differ from the images you attached: those images are intensity images only. I'll have a look at png's and float values.

Best, Patrick

[FabianIsensee]

I would probably avoid png images when dealing with floating point (and potentially negative) values ;-)

[FabianIsensee]

https://www.mathworks.com/help/images/ref/niftiwrite.html