search

got-10k / toolkit

Official Python toolkit for generic object tracking benchmark GOT-10k and beyond
http://got-10k.aitestunion.com/
MIT License
557 stars 95 forks source link

Are there any plans adding EAO calculating to VOT performance? #14

Open LCAR979 opened 5 years ago

LCAR979 commented 5 years ago

Thanks for your beautiful work. Testing on VOT now is quite easy and we no longer need to deal with those bugs in code of vot. Just open this issue to ask, do you have any plans to add EAO calculation in the short future? Also the robustness score may also need to be adjusted.

I have tried to use the results generated by your code, only ploting EAO ranking using official version. However, lots of bugs appeared.

huanglianghua commented 5 years ago

Hi, the IoU calculation error of VOT-toolkit has been fixed recently (Issue 202), meaning the IoU calculation of our toolkit and VOT becomes consistent.

Yes, we'll add the EAO code (and probably other metrics) in a near future. Please stay tuned. Thanks.

jonathantompson commented 5 years ago

+1. I've had a lot of trouble getting the official toolkit working in our system due to the use of Matlab (and mex files in particular). This is a great stab at improving the portability of VOT eval! Thanks so much for putting this awesome library together.

The unsupervised metrics would be much appreciated. Looks like the realtime and unsupervised experiments are already complete, but the report is missing the metric calculation. Is this correct?

huanglianghua commented 5 years ago

@jonathantompson Yes. Currently one can direct generate baseline/realtime/unsupervised results using this toolkit and evaluate performance using VOT-toolkit. The recorded results are compatible.

jonathantompson commented 5 years ago

@huanglianghua

Quick question. A lot of authors report supervised robustness using the "failure rate" metric. As an example see: https://arxiv.org/pdf/1812.11703.pdf. Is this quantity relatively easy to calculate in your framework?

Again, thanks for putting together such an awesome repo.

rgoroshin commented 5 years ago

Is "robustness (failure rate)" even reported by the VOT toolkit as a number (and not just a point on the AR-plot)? If so can you point me to where I can find it? I only see Robustness as a weighted mean number of failures in my VOT report. Thanks!