computing MSE metric

[alalbiol]

Hi I have just read your neurocomputing paper and found it very insteresting. I have one question about how do you compute the MSE metric in the paper. As I undestand your method may detected a different number of landmarks, in this case how do you select the appropieate landmarks to calculate MSE, and what do you do if one landmark is missing (not detected) ? I have looked into the code but I can't find how do you compute the metric

Thanks

[zc402]

Hello, Thank you for paying attention to my project.

The reason of using MSE index is that we want to be consist with paper [1] by dataset provider, so as to make comparisons. According to [1], 1 pixel of bias in an 1000x1000 image causes 0.01 unit of error, which is $0.01 \times 0.01 = 0.0001$ MSE. Therefore, if the spinal image is stretched to 1000x1000, then our predicted landmarks (0.0039 MSE) are averagely 6.24 pixels away from ground truth.

To compute MSE: firstly stretch the result image into 1000x1000 pixels, then compute error in pixels: e = mean_pixel_distance(gt, pred), then convert to MSE: (e * 0.01)^2.

The MSE evaluation index is not very intuitive, I do not know why paper [1] chose to use MSE instead of distance-based index.

About the missing landmarks, since we only apply MSE evaluation to the test images in easy samples ("50 tight images" in our paper), there was no missing landmark. Some times extra landmarks are detected, in that case we compute the distance of pred landmark to nearest ground truth landmark. I know that sounds cheating, but I believe that paper [1] also did that: I mentioned a wrong alignment problem in fig.9 in my paper, where fig.9(a) is borrowed from paper [1]. Red rectangles marked wrong alignments, which was caused exactly by "missing landmarks". But the their result shows that test set pred landmarks are averagely 6.78 pixels away from ground truth, which is not possible if it was not compared with nearest ground truth landmark. Anyway, I think that MSE is not a good metrics for this scoliosis detection task, the SMAPE is a better choice.

Thanks

[1] H. Wu, C. Bailey, P. Rasoulinejad, S. Li, Automatic landmark estimation for adolescent idiopathic scoliosis assessment using boostnet, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2017, pp.127–135.

[alalbiol]

Thank you very much for your quick and very honest answer. I totally agree with all your comments and I can imagine you having to add the MSE evaluation to make one of your paper reviewers happy :-D

[zc402]

Exactly! I'm glad that someone understands me :)