EAO in vot2018

[phiphiphi31]

I like your project very much. But I test your tracker using pysot toolkit. The average eao ( 10 times )is around 0.46 which is lower than 0.489. I wonder if there are the differences of hardware or toolkit or not affectcting the results. Which toolkit do you use to evaluate your tracker?

[alanlukezic]

Tracker was evaluated using Matlab version of the VOT toolkit and the results should be deterministic, so it is strange that they differ across multiple runs. We observed that there can be slight differences in tracking performance depending on hardware and library versions (pytorch, torchvision, cudatoolkit, etc.)

[phiphiphi31]

I think the official vot toolkit calculates the average number of 15 tests so it is normal that they are different across multiple runs. I also test the Atom and Dimp50. The average EAO in ten times is similar to their results in paper. ![Uploading dimp.png…]()

[phiphiphi31]

the average eao of dimp is around 0.44, atom: 0.40.

[phiphiphi31]

could you send me your raw results (txt files) from matlab toolkit version ? my email 372998044@qq.com , thanks a lot!

[phiphiphi31]

By the way, I appreciate your work D3s very much, because it achieves a high EAO with only youtube VOS training dataset and uses only one single GPU. It is much better than the siamese tracker which requires more training resources. Moreover, it bridge the gap between segmentaion and tracking which is awesome !

[alanlukezic]

The results are obtained by running the algorithm only once since we have modified the existing ATOM code to produce deterministic results. Therefore the whole algorithm is deterministic and there is no need for running it multiple times. The preliminary results for VOT18 can be found here: http://data.vicos.si/alanl/d3s/D3S.zip

[phiphiphi31]

thanks a lots ! probably the cuda toolkit or cudnn affect the testing results

[lawpdas]

@alanlukezic Hi! How to make the ATOM/DiMP tracker produce deterministic results? Could you give me some suggestions?

[deepbrainme]

@alanlukezic , Thanks for your amazing work. Would you share some experience about how to repreduce your result performance writed in your paper such as (EAO:0.489). I was confused when I test your code for dramatic ups and downs ,because only test one time is not deterministic. Looking forward your reply.

[sherlockedlee]

Same problem as yours.The results I produce using pysot is 0.463.

[phiphiphi31]

atom and dimp cannot product deterministic results because it uses random bbox candidates in IOU net. If you only use filters without IOUnet, it is deterministic. Also, you need to set: torch.cudnn.benchmark = flase.

[deepbrainme]

It seems reasonable. However I was still confused for the random results in my experiments . Did you get the stable result up to 0.489 in average 15 times? I will be appreciate if you cound show me some result about this paper in vot2018.

[alanlukezic]

From my experience, the results can vary slightly depending on the hardware and versions of the python libraries. My setup is the following: Ubuntu 16.04, NVidia GeForce GTX 1080, Python 3.7.4, PyTorch 1.1.0, Cuda 9.0.176, cudnn 7.5.1, torchvision 0.3.0, cudatoolkit 9.0.

The results were obtained by a single run using VOT Matlab Toolkit. Running the tracker multiple times produces identical results every time, therefore running the tracker multiple times and then averaging the results does not make sense.

[deepbrainme]

Thanks for your nice work . I came across some questions. Here are my environment. Ubuntu 18.04, NVidia Titan XP， Python 3.7.7， PyTorch 1.1.0, Cuda 9.2.148, cudnn 7.6.4, torchvision 0.3.0, cudatoolkit 9.0 The envs is almost same as yours .

Here is my result by running once on vot2018 benchmark . EAO 0.470，ACC 0.658 ，ROB 0.178 ，lost 38 It is far from your result claimed in your paper 'D3S'.Is this reasonable? I'm comfused for the result far from yours. That will be more convincing and if other reader is able to reproduce your result with same Hardware and software.Would you share your experience? Thanks a lot.

Another thing : The official matlab vottoolkit tool is running 15 time by default. Only run once maybe not be reliable even if your tracker is deterministic . Can you reproduce your result by running 15 times in your computer with matlab vottoolkit for a fair comparison with other SOTA tracker. Just run a few hours for the fair comparison. Looking forward your new raw results.

[alanlukezic]

I re-run the experiments once again using VOT Matlab toolkit (versioin 7.0.2) and it automatically finishes the experiment after third run on each sequence since it detects that the results are identical which means that the tracker is deterministic. The results on vot18 are the same as reported in the paper.

[zhanght021]

atom and dimp cannot product deterministic results because it uses random bbox candidates in IOU net. If you only use filters without IOUnet, it is deterministic. Also, you need to set: torch.cudnn.benchmark = flase. @phiphiphi31 Hello, I got the random results like you, and I set torch.cudnn.benchmark = flase in the beginning of the code, but the results of D3S is also random, so if there any other place should I modify? Thanks.

[phiphiphi31]

atom and dimp cannot product deterministic results because it uses random bbox candidates in IOU net. If you only use filters without IOUnet, it is deterministic. Also, you need to set: torch.cudnn.benchmark = flase. @phiphiphi31 Hello, I got the random results like you, and I set torch.cudnn.benchmark = flase in the beginning of the code, but the results of D3S is also random, so if there any other place should I modify? Thanks.

For d3s , cuddnnbencmark = false. Projection matrix init = pca, filter init = zeros. Dont use randm and msra to init. And IOUnet can not be determinstic if you use it.

[zhanght021]

@phiphiphi31 Thanks a lot, I'll try again.

[bigonestep]

@phiphiphi31 Hello, can you provide me with a method to use the pysot toolkit to test d3s? My attempt to transplant failed.

[alanlukezic]

The method was evaluated using the VOT toolkit and evaluation scripts from pytracking repository, which are included. I did not use pysot for evaluation, but maybe someone else did.

[hekj]

Hi, friends. I wonder how you get your results. Have you use the given pretrained model or the model trained by yourself on the dataset listed in the paper.

• Firstly, I test the given model on VOT2018 and OTB100. The results are as follows VOT2018 OTB100

• Then, I test the model trained by myself following the paper says on VOT2018. The results is as follows

The results seems to be a little terrible. I don't know the reasons and what I should do to improve the performance. Can someone tell me about that? Thank you!