search

votchallenge / toolkit

The official VOT Challenge evaluation and analysis toolkit
http://www.votchallenge.net/
GNU General Public License v3.0
153 stars 43 forks source link

Cannot get 100% accuracy despite cheating #63

Open CodeOfDuty44 opened 1 year ago

CodeOfDuty44 commented 1 year ago

I tried to get 100% accuracy out of curiosity. For this reason I created a fake sequence of length 10 and set the anchor for the first frame only. I then performed "vot evaluate myTracker" and got the results. Result txt consists of predicted bbox values, except for first line, which is "1" instead of "x1,y1,w,h" values. Then I changed every line except for the first line to the corresponding groundtruth values, that is to say, I tried to cheat and get 100% accuracy.

However I got 0.9 accuracy. Suspecting first line related issues, I did the same for fake sequence of length 20, and I got 0.95 accuracy this time. It seems that vot toolkit also expects first line of generated result file to be the corresponding groundtruth as well. But this requires me to run my tracker on the frame I used to initialize it, which is a little bit strange and does not quite make sense. Do I need to generate and report the result for the initialization frame as well, or is this some kind of bug? Do I miss something?

Thanks in advance.

lukacu commented 1 year ago

Hm, thanks for this "cheating", it is indeed strange that you cannot get 100% score. We will check it out and let you know.