Random Intensity transforms for test-time augmentation

[han-liu]

Hi,

Test-time augmentation is a helpful tool to use but it seems that currently only spatial transformations are included for TTA. Intensity transformations such as Random contrast are not invertible transforms and thus cannot be used in current TTA implementation. Is there a way that I can use intensity transforms as TTA? Thanks.

[Nic-Ma]

Hi @rijobro ,

Could you please help share some comments about the TTA plan here?

Thanks in advance.

[rijobro]

Hi @han-liu we certainly have it on our agenda to extend invertible transforms to include more than just spatial ones. Which intensity transform in particular are you using? Perhaps I can make this one invertible to be used as an example for future intensity transforms.

[tvercaut]

I guess invertible transforms don't have to be necessarilly mathematically invertible. We just need to define a suitable pseudo-inverse for it.

The example of adding Gaussian noise on the image intensities is interesting to consider. We could either make it an invertible transform by storing the noise added at test-time or just forget about the noise values and consider that the pseudo-inverse is the identity mapping. In any case, if TTA with Gaussian noise on the image intensities is to be used for segmentation purposes, the noise values are irrelevant (just a nuisance paraneters) for the segmentation output. In this case the pseudo-inverse to be applied to the network output is the identity.

Do we already handle different specialisation of the inverse transformation based on the type of network output?

In terms of really non-invertible transforms (those where storing the parameters is not enough to revert it up to numerical issues), I don't have a realistic example in mind. However, I guess that considering an image intensity change transform whereby we take the absolute value of the input intensity is a sufficient test case. In this scenario, we might want to have no defined inverse if the network is an auto-encoder but consider an identity pseudo-inverse if the network output is a segmentation map.

By the way, this raise the question of what to do for networks which are note image-in-image-out, e.g. classifiers. There as well, we might need to specialise the definition of inverse.

So in short, we should probably include some knowledge of the codomain and whether the transform acts of the input domain and output codomain in an equivariant fashion.

[han-liu]

@rijobro @tvercaut Thanks for the clarification and I think what @tvercaut proposed here is a great feature to be added to TTA.

[rijobro]

@tvercaut to take the example of GaussianNoise, the inverse of the spatial component of that transform is identity, because obviously adding Gaussian noise doesn't modify the image spatially. Having said that, I think it would be complicated for us to therefore say that a pseudo of the inverse can be given as identity. Because at that point, we're saying that the most important component of the inverse is the spatial component. This may or may not be true, but I'm not sure we really want to make a decision either way.

A potential solution might be to have something like:

def inverse(self, x):
    x = self.spatial_inverse(x)
    x = self.intensity_inverse(x)
    # any other types of inverse
    ...

But I think this starts to get messy and is probably overkill for what we want. And no, we don't have different inverses depending on the network type.