Investigate the relevance of DA for addressing out-of-distribution contrast images

[jcohenadad]

From this presentation, results below, the first image is from spine-generic (good prediction), and the four others are not (bad prediction):

We need to understand how is the contrast of the image that produce bad prediction, and how DA could be improved. The purpose of this issue is to document how the DA affects image. For example, you could show us examples of DA images.

Code version: https://github.com/neuropoly/totalsegmentator-mri/blob/retrain-with-sc-and-sacrum/src/totalsegmri/utils/generate_augmentations.py

[yw7]

To better understand how the contrast of the images affects the prediction quality and how data augmentation (DA) could be improved, we should investigate how the current DA methods impact the images.

To address this, I propose a new segmentation-based augmentation approach for each class (spinal cord, canal, vertebrae, intervertebral discs):

1. Calculate the mean and standard deviation of the intensity values for the class
2. Adjust the standard deviation by multiplying it by a random factor between 0.9 and 1.1
3. Compute the probability density function (PDF) for the class based on the adjusted mean and standard deviation
4. Apply the PDF to the image, multiplied by a random factor between -1 and 1

The augmentation will then sum the results from step 4 for all classes, normalize to range 0-1, multiply by a random factor between -1.3 and 1.3, and add to the original image.

Illustration of the proposed method:

Results (the last column is the original image, each row is a different subject):

For comparison, I will also show the results of current augmentation based on random gamma, log, square root, exponential, sinus, and sigmoid transformations (the last column is the original image, each row is a different subject):

I expect that inferring the model on the images augmented with the new segmentation-based approach will yield different results compared to the original images, while the gamma, log, sqrt, exp, sin, sig augmentations will produce results more similar to the original. This would suggest the new approach alters the relationships between the intensity values of the different anatomical classes in the augmented images.

Model inference on the new proposed augmentation:

Model inference on current augmentation based on random gamma, log, square root, exponential, sinus, and sigmoid transformations: