search

hkchengrex / Mask-Propagation

[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.
https://hkchengrex.github.io/MiVOS/
MIT License
127 stars 22 forks source link

Question about result on DAVIS16 #33

Closed yongliu20 closed 3 years ago

yongliu20 commented 3 years ago

I want ask why I get this result using your pre-trained model? Thanks! image

hkchengrex commented 3 years ago

That doesn't look right... Can I see some output images?

hkchengrex commented 3 years ago

Specifically, you can compare them with our pre-computed results.

yongliu20 commented 3 years ago

The results of 'blackswan' are these: image

yongliu20 commented 3 years ago

I also think this J&F result is not right but I can not find out what went wrong...

hkchengrex commented 3 years ago

"blackswan" will almost always be good because it is so easy... Can you download our pre-computed results and see if there are any differences? I want to check whether the problem lies in generation or evaluation.

yongliu20 commented 3 years ago

Ok, I will do it now

yongliu20 commented 3 years ago

There are indeed some differences between them but very small. And I test your pre-computed results, the J&F is: image The evaluation method which I use is from davis.

yongliu20 commented 3 years ago

I find that my result is as same as the D16_s012. And the above result belongs to D16_s012_notop.

hkchengrex commented 3 years ago
  1. I just evaluated again and I got the same reported number (90.9 for D16_s012_notop). To check that my evaluation code is correct, I downloaded STM's results (https://github.com/seoungwugoh/STM) (mapped from 0\~1 to 0\~255) and evaluated them in my environment. I got the same result as the reported number in STM. So most likely your evaluation has some problems.
  2. Visually, the results also look closer to 90 than 75.
  3. I also found that I swapped two rows (D16_s02, D16_s012_notop) in the table in README. I will fix that, thanks for the hint.

DAVIS 2016 evaluation code is not very well maintained. It was not easy to get it right for me back then... I recalled there are some discussion threads about a proper implementation. I am going to check on those and get back to you.

hkchengrex commented 3 years ago

Here: https://github.com/davisvideochallenge/davis2017-evaluation/issues/4 Hope it helps.

You can always check the numbers with ours/STM's. I might open source my own evaluation code later.

yongliu20 commented 3 years ago

Well, I have solved this problem by modifying your eval_davis_2016.py, like this: image

hkchengrex commented 3 years ago

Well yeah if your evaluation script expects 0/1 outputs... The ground truths in DAVIS 2016 are 0/255 so I'm sticking with that. Glad that it has been fixed.

hkchengrex commented 3 years ago

Hmm I think I can actually modify the code a bit to make both happy. Gonna do that. Thanks.

yongliu20 commented 3 years ago

My previous problem is that the output values of the foreground pixels are not same. And thanks for your help!