Sliding window inference with overlap and weighting

[jenspetersen]

Is your feature request related to a problem? Please describe. Image-scale predictions (e.g. segmentations) tend to become worse towards the outside of the image [citation needed, I know], resulting in non-smooth predictions when a sliding window approach is used.

Describe the solution you'd like An option to perform sliding window inference with overlapping patches (e.g. by half patch size) and weighting (e.g. a Gaussian over the image) so we put more weight on predictions from the center of a patch.

[Nic-Ma]

Hi @jenspetersen ,

Thanks for your feedback! That's a very good point. Actually, we already discussed internally about overlapping in sliding window, and want to have it in a future version. I think that's a very important and valuable improvement to our evaluation method. Do you have any interest to submit a PR that enhances our existing sliding window method? Thanks.

[jenspetersen]

@FabianIsensee, you already implemented this, is that something you would/could contribute?

[FabianIsensee]

Hi, yes I already did that a while ago and I actually rewrote the whole thing a couple of days ago because it was not easy to read and quite convoluted. At the moment I am very busy and, as much as I would like to, I cannot contribute in the near future. In case someone is interested, here is my implementation:

https://github.com/MIC-DKFZ/nnUNet/blob/master/nnunet/network_architecture/neural_network.py

The code for sliding window prediction in 3d is here:

https://github.com/MIC-DKFZ/nnUNet/blob/dba763e1abbb311f8f9fbed842ff9bde7dce87e7/nnunet/network_architecture/neural_network.py#L269

2d is here:

https://github.com/MIC-DKFZ/nnUNet/blob/dba763e1abbb311f8f9fbed842ff9bde7dce87e7/nnunet/network_architecture/neural_network.py#L589

Best, Fabian

[Nic-Ma]

Hi @madil90 ,

Could you please help develop this feature request? It will obviously improve the evaluation performance. Thanks.

[madil90]

@Nic-Ma Yes definitely. I'm looking at @FabianIsensee implementation and will start porting it to MONAI.

[mjorgecardoso]

@Nic-Ma @madil90 Getting the patch weights with Gaussians is only one way to do this, but we should probably support other approaches. For example, the next two immediately come to mind.

• Option 1: Get the window stride (overlap) to be equivalent to the maximum receptive field of a network. This one will be very harsh at rejecting any prediction close to the border of a patch, but will guarantee that the predictions are always valid by construction.
• Option 2: Get the mixing weight to be something else other than a Gaussian (e.g. distance from the centre of the patch). Given that kernels are 3x3, information propagation along the network does not "travel" isotropically, and thus should not be Gaussian (it is actually the Chebyshev Distance).

Also, we need to be aware that not everything is a semantic segmentation model. E.g. weighted average of 2 predictions is not necessarily ideal for semantic regression models (image synthesis).

[madil90]

@mjorgecardoso Thanks, that's a very good point. I'll separate the weighting from overlapping in a way that we can implement other weighting algos later.

[Nic-Ma]

@mjorgecardoso Thanks, that's a very good point. I'll separate the weighting from overlapping in a way that we can implement other weighting algos later.

Hi @madil90 ,

Thanks for your help on this PR! Looking forward to your update of our sliding window.