search

votchallenge / toolkit-legacy

Visual Object Tracking (VOT) challenge evaluation toolkit
Other
381 stars 167 forks source link

Why the results are different? #121

Closed yingjieyao closed 3 years ago

yingjieyao commented 7 years ago

Sorry to bother but I don't know why the results of VOT2015 I get are not consistent with the VOT2015 Results. I download the VOT2015 results from the official site and run the toolkit to generate the report, but happen to find the result I get is not consistent with the paper states. My toolkit version is major 4, minor 1.

Another finding is that all my results are consistent with some papers, such as SiameseFC.

Anything else I need to do to generate the VOT2015 report?

For example, the MDNet, EBT, DeepSRDCF and SRDCF's results all seem lower the than the results in the vot2015 paper with respect to accuracy.

lukacu commented 7 years ago

Sorry for late reply, I somehow missed this issue. Can you provide an example of a difference (what is its scale)? One of the reasons for small differences may be that we are constantly changing the toolkit which unfortunately also introduces some changes in analysis. To completely reproduce the results you may then try to use the same toolkit that was used for that challenge (in your case something like 3.1), but since we do not provide builds for older version you may have to compile it yourself.

gongbudaizhe commented 6 years ago

I have encountered similar issues #148

For example, the non-realtime Accuracy of KCF is 0.4339 but the reported value is 0.447. Robustness and EAO is the same though. I have also tried the SiamFC tracker, the Accuracy is consistently lower than the reported value in VOT17.