search

Jee-King / CVPR2022_STNet

MIT License
86 stars 19 forks source link

The RSR and RPR are inconsistent with your results. Could you upload your metrics evaluation code? #7

Open wshku opened 2 years ago

wshku commented 2 years ago

Using your code of test.py and test dataset of FE240hz and VisEvent, I get your tracking results. Then I use the metrics evaluation code from the VisEvent project, https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark, to evaluate your STNet's performance. I went through the VisEvent code and it is fine.

Here are my results: FE240hz: RSR 60.3%, RPR 82.3%, VisEvent: RSR 26.3%, RPR 49.9%,

some of which are inferior to your claimed results in the paper: FE240hz: RSR 58.5%, RPR 89.6%. VisEvent: RSR 35.5%, RPR 49.2%.

STNet seems not to outperform the counterpart methods. I am wondering whether you could provide your evaluation code

laisimiao commented 2 years ago

@Jee-King Could you provide your raw results on FE240hz and VisEvent so that we can evaluate them directly on their toolkits?

Jee-King commented 1 year ago

Thank you for pointing it out. We evaluate all trackers by following pytracking. We checked the checkpoint.pth we uploaded, and the performance on FE240hz is: RSR 60.2%, RPR 87.9%. In fact, the uploaded checkpoint in GitHub is slightly different from the camera-ready version. For example, only 48 non-rigid sequences in the Visevent are used for testing in this code, but 172 sequences are used for testing in our camera-ready.
Therefore, there is a gap in the results on VisEvent. You can download our raw results of the paper from here. You can also see all the VisEvent test sequences we use.