search

visionml / pytracking

Visual tracking library based on PyTorch.
GNU General Public License v3.0
3.19k stars 603 forks source link

Invalid frames handling on datasets #391

Closed sfchen94 closed 1 year ago

sfchen94 commented 1 year ago

Hi, Thanks for your work, I am curious why some datasets handle invalid frames when reporting the evaluation results, but others do not. For example, there exist different evaluation results when we handle the Invalid frames on LaSOT. But most of your work skips to handle invalid frames on this dataset. Is there any reference to explain that?

2006pmach commented 1 year ago

Unfortunately different papers/datasets came up with slightly different implementations of the same metric. Thus, there are some small differences such as including or excluding invalid frames depending on the benchmark. In the end, we made sure that the results produced by our implementation can reproduce the results reported in the individual papers (LaSOT, OTB, NFS, UAV etc). Maybe we should not expose this to the user and simply select these parameters automatically depending on the datasets that is used for evaluation...