resolution independence ... pushing the limits

[romainVala]

Hi

Thanks for providing this nice tool. I was very interested in your VINN paper, the resolution independence is really an important and unique feature. I must admit that I was doubtful that it would work. May be I do not fully understand, but it looks like you just added a reslicing within the network (instead of doing prior on the data and after on the prediction, I need to compare, but I expect that sinc interpolation (on the 4D labels) should already do a pretty good job

You say in the paper that it avoid a loosy interpolation, but you still perform an interpolation at the begining of your VINN network, (and this interpolation is not learned by the model, you use fixe parameter right ?) may be the pre-IDB convolutions, helps to learn a better interpolation ?

At the end of the day nothing better than a real test. I get this crasy dataset from openneuro (https://openneuro.org/datasets/ds003563/versions/1.1.0) which provide a T1 in-vivo acquisition at 0.25 mm³ ! So I tested with a resampled version at 0.5 mm and the results are very nice (you still get a bit of dura matter in the cortical gray matter, (GM) ) but the results are much better compare to what I get pour spm cat12 segmentation. I do find it an amazing result espetially because the model never see resolution data below 0.7 When I test with 0.25 resolution, at least it goes through it (spm cat12 just crashed), but then the segmentation is bad (a lot of missing GM) . The contrast and SNR is probably a bit too low but also we are facing the method limitation here ?

I am curious to know your thought about what is the bottleneck if we want to target such low resolutions . Should we for instance increase the inner resolution of the model (currently 1mm, because your targeting 0.7 mm) ? or will it be more a problem of training samples ?

Romain

[dkuegler]

Hi, Our FastSurferVINN paper had some results with respect to generalization to resolutions outside of the training data. Figures 7 or 8 I think.

Generally, we hope the method would work at around 0.5mm, we are quite confident it works for 0.7+. But the further you reduce the voxelsize, the less this will work. It has just never seen something similar and thus may use invalid assumptions about the structure of the data

So, I would have been surprised if it worked for 0.25mm. You just get so much more detail at that resolution, that images just look very different in the local neighborhood, e.g. blood vessels. Low SNR has a similar effect here.

Also, it seems there are only very few images available right now, so training is difficult and does not currently have a large community....

At this point, I would assume training data (training corpus) to be the most important bottleneck, but the range of target resolutions may also contribute to limited performance... so there may also be a methodolical limitation here.

[dkuegler]

To the interpolation questions, we tested multiple kernels and linear was best. Maybe sinc would also work, but would not be efficiently supported by pytorch.

Generally, VINN does 2 interpolations -- but both of them are in latent space. The "learned" part here is not referencing the fact that we learn how to do the interpolation, that is indeed static (only parametrized by the voxel size), but we learn "which features" work well for interpolation. That is a critical feature as shown by the ablation in our paper (+inSA).

[romainVala]

thanks for your insightful comment. The 0.25mm has more detail, but we see the same structure compare to 0.5 (the 0.5 is just a resample version) and there is not new anatomical structure. but ok I see your point, as precise in the doc, below 0.7 is experimental, but quite good up to 0.5 : nice