Problem when running inference on data with different intensity ranges

[vcasellesb]

Hi,

I'm gonna try and do my best explaining the issue, and posing the information I've gathered in a coherent manner.

My problem is that I get empty masks when trying to segment CBCT (CT) images that have a different range of intensity values to the range of my training dataset. I ran training on a dataset of N=130. Each image had a range of values of (-1000 -- 5000). So it had negative values, and max intensity values of between 3000-5000 approx.

Then, I was trying to predict images that have a range of values between 0-2000. So no negative values and a much smaller range. The predictions came up empty. I tried "transforming" this new images using the following formula:

This worked, I got pretty good segmentations.

Then, I tried to circumvent this issue by scaling (using the same formula as above) all training images to a range of [0, 2048]. Then, during inference, input images also would be transformed to that range. However, this method has proven to work way worse than the first one.

I've been diving into the code, and since I've specified that the channel for my images is "CT" (dataset.json file), a normalization scheme is performed by the nnU-Net code (namely, a subtraction of the mean and dividing by the deviation). Therefore, I would like to ask the following: why, given that the nnU-Net code runs normalization already, do my new-range cases (the ones I mention at the third paragraph) get an empty mask? Are images not "normalized" during inference?

Then, and I don't know if this is the appropiate forum for this topic, and given my limited knowledge of deep learning: Does it make sense to scale the images myself (using the formula in the screenshot, as I've mentioned above) to a range of [0, 2048], given that nnU-Net will perform normalization automatically? Could that be the case of me getting worst results on my second iteration of models (with the scaling to [0, 2048] performed)?

Thank you very much for your time, and if these questions have already been posed or they don't belong here, please let me know.

Best regards, Vicent Caselles Ballester

[ManuBN786]

I faced this problem a few days ago !!

The model was trained on the Med Decathlon dataset which has intensities in the range (-1024, +3071) And then I tried to predict on LUNA16 which had intensities in the range (-3024, +3071) and LIDC-IDRI with range 0f (0,255). But got all blank images.

So I converted all the datasets to .png first using:

test_load = nib.load(nii_img_path).get_fdata()

if test_load.shape[2] > 0: for ii in range(test_load.shape[2]): test = test_load[:, :, ii] plt.imsave(writepath + name + '' + str(ii) + '.png', test, cmap='gray')

And then I used 'nnunet/dataset_conversion/Task120_Massachusetts_RoadSegm.py' to convert all the .png to .nii files.

This normalized the intensities of 3 datasets into one. Then I trained the model again and the prediction was perfect.

I'm not sure if this is the right way, but it worked for me !

[vcasellesb]

Hi Manu,

Thanks for your reply. There are some things I don't understand about the code you shared.

First, at the if chunk when you check if test_load.shape[2] > 0, I guess you are checking if the image is three dimensional and if it is you are slicing the image along the 'z' axis, right?

Secondly, I don't see where you transform the intensities anywhere in your code? Can you clarify this?

Thank you very much.

Best regards, Vicent Caselles

[ykirchhoff]

Hi @vcasellesb, @ManuBN786,

for CT nnUNet applies a normalization scheme, where intensities are normalized according to the extracted foreground intensity statistics, see https://github.com/MIC-DKFZ/nnUNet/blob/b4e97fe38a9eb6728077678d4850c41570a1cb02/nnunetv2/preprocessing/normalization/default_normalization_schemes.py#L52. This is done because for CT the Houndsfield Units are quantitative values, i.e. certain structures belong to specific intensity ranges. If you experience shifts in the intensities between your datasets, I would suggest to set the modality to something else and it will apply z-score normalization, see also here, which you also apply to e.g. MRI. I wouldn't fiddle around with the intensities too much apart from that, as this can just result in additional shifts if you don't take the distribution into account.

Best, Yannick

[vcasellesb]

Hi @ykirchhoff,

Thanks for your reply. This poses another question. What kind of preprocessing is performed when trying to predict new images (i.e. when there is no label)? Which information is used to perform the Z-scoring?

Sorry if this is very obvious, but I've been looking at the source code for days and cannot understand which kind of preprocessing is performed.

Thanks again for your support and work.

Best, Vicent

[ykirchhoff]

Hey Vicent,

nnUNet will (usually) use a nonzero mask for normalization and cropping of the images. This mask is determined by the outside region, which is typically at a value of zero, at least for MRI (see here). All values inside this region will be used for normalization of the scans, so no label information is needed for z-scoring.

Best, Yannick