Reproducing scores

[simplerick]

Hi, what IoU score to expect at the end of the evaluation? I got about 0.5 when running flowpsam on the dvs17 dataset at 480p resolution

[Jyxarthur]

Thanks for your interest in our work. The (framewise) IoU performance of flowpsam on DAVIS2017 should be around 69%. Would you mind sharing more details about the ckpt you adopted, and other hyperparameter settings (e.g., flow gaps, image encoder for both rgb and flow inputs)?

[simplerick]

Ah, I found my mistake, used the checkpoint for davis17m. The results are confirmed, but why does the metric drop so significantly when I change the checkpoint?

I wonder how well the method generalises to other data. What checkpoint should I use for custom data?

[Jyxarthur]

Yes, the DAVIS17 and DAVIS17-m ckpts could be quite different, as these datasets adopt exactly the same RGB images, but with different annotations (DAVIS17-m groups the commonly moving objects as a whole). This results in a performance discrepancy. For general application, I suggest using DAVIS17 (w/o -m) ckpt, which is also generalized to other datasets such as SegTrack, FBMS, and YTVOS-m.

[simplerick]

Thank you for your work and advices! I will close the issue as it's resolved