search

hustvl / Matte-Anything

[Image and Vision Computing (Vol.147 Jul. '24)] Interactive Natural Image Matting with Segment Anything Models
MIT License
465 stars 33 forks source link

About evaluation on P3M500 #1

Open felix-ky opened 1 year ago

felix-ky commented 1 year ago

Hi, thnaks for releasing the great repo and paper. I evaluate matteformer and vitmatte (both using pretrained weight) on P3M500, but both model's MSE error (times 1e3, computed on unknown areas) are 10 times of your result in Table 4, so are you scaling 1e2 for mse error in unknown areas? image

JingfengYao commented 1 year ago

Hi, thnaks for releasing the great repo and paper. I evaluate matteformer and vitmatte (both using pretrained weight) on P3M500, but both model's MSE error (times 1e3, computed on unknown areas) are 10 times of your result in Table 4, so are you scaling 1e2 for mse error in unknown areas? image

Hi, thanks for your question.

The evaluation code given by ViTMatte evaluates MSE on only trimap unknown region. However, since the methods in Table 4 are not all trimap guided, it is unfair to evaluate in that way. So our results are evaluated on the whole image.

As ViTMatte for instance, for same the inference result, MSE would be 13.17 on unknown regions and be 1.24 on the whole image.