Reproducibility of results

[UmarSpa]

Hi

I trained the model from scratch using the instructions you provided (link). I can see the training setting in the yaml file (link) is the same as mentioned in the paper. The training script seems to work fine, but when I test the trained model, I don't get the same results as reported in the paper.

I run the training using following command:

python tracking/onekey.py

The training curve I obtain is as folllowing:

I run the testing using the checkpoint form the last epoch:

python tracking/test_ocean.py --arch Ocean --resume snapshot/checkpoint_e50.pth --dataset VOT2019

I run the evaluation on the obtained testing results:

python lib/eval_toolkit/bin/eval.py --dataset_dir dataset --dataset VOT2019 --tracker_result_dir result/VOT2019 --trackers Ocean

The results I obtain are as following:

Tracker Name	Accuracy	Robustness	Lost Number	EAO
Ocean	0.602	0.426	85.0	0.283

As you can see they differ quite a lot from the results reported in the paper (Accuracy: 0.590, Robustness: 0.376, EAO: 0.327), and also from the results I obtain using your provided checkpoint. Is there any explanation for this ?

[JudasDie]

Hi

I trained the model from scratch using the instructions you provided (link). I can see the training setting in the yaml file (link) is the same as mentioned in the paper. The training script seems to work fine, but when I test the trained model, I don't get the same results as reported in the paper.

I run the training using following command:

python tracking/onekey.py

The training curve I obtain is as folllowing:

I run the testing using the checkpoint form the last epoch:

python tracking/test_ocean.py --arch Ocean --resume snapshot/checkpoint_e50.pth --dataset VOT2019

I run the evaluation on the obtained testing results:

python lib/eval_toolkit/bin/eval.py --dataset_dir dataset --dataset VOT2019 --tracker_result_dir result/VOT2019 --trackers Ocean

The results I obtain are as following:

Tracker Name Accuracy Robustness Lost Number EAO Ocean 0.602 0.426 85.0 0.283 As you can see they differ quite a lot from the results reported in the paper, and also from the results I obtain using your provided checkpoint. Is there any explanation for this ?

Hi, thanks for your interest. I think you (may) haven't carefully read the tutorial and onekey scripts. You may test epoch 30-50 and then select the one with best result (Test in onekey). Then, tune the hyper-params (Tune in onekey).

[UmarSpa]

I see. I used the default values in onekey script. I will change the Testand Tune flags in onekey now, and rerun the experiments. Thanks for the prompt response.

[UmarSpa]

Hi. I am still working on this. Now the problems seems to be with ./tracking/tune_tpe.py. When I run it, I get the following error:

Traceback (most recent call last):
  File "./tracking/tune_tpe.py", line 43, in <module>
    tracker = Ocean(info)
  File "/tracking/../lib/tracker/ocean.py", line 17, in __init__
    self.online = info.online
AttributeError: 'EasyDict' object has no attribute 'online'

In fact when I look into info, I have only these values:

{'align': True, 'arch': 'Ocean', 'dataset': 'VOT2019', 'epoch_test': False}

Any idea what might be wrong here ?

[JudasDie]

Hi. I am still working on this. Now the problems seems to be with ./tracking/tune_tpe.py. When I run it, I get the following error:

Traceback (most recent call last):
  File "./tracking/tune_tpe.py", line 43, in <module>
    tracker = Ocean(info)
  File "/tracking/../lib/tracker/ocean.py", line 17, in __init__
    self.online = info.online
AttributeError: 'EasyDict' object has no attribute 'online'

In fact when I look into info, I have only these values:

{'align': True, 'arch': 'Ocean', 'dataset': 'VOT2019', 'epoch_test': False}

Any idea what might be wrong here ?

Have you updated to the new version?

[UmarSpa]

I have updated it now. Hopefully all goes well now.

Out of curiosity can you share the EAO of your best model (Ocean offline) on VOT2019 after running ./tracking/test_epochs.py on checkpoints from 30 to 50 epochs ?

The value I get is 0.297 (EAO), and I was wondering if its reasonable. And that after I tune it with tune_tpe.py should I expect it to reach somewhere near 0.327 (EAO), as reported in the paper.

[JudasDie]

I have updated it now. Hopefully all goes well now.

Out of curiosity can you share the EAO of your best model (Ocean offline) on VOT2019 after running ./tracking/test_epochs.py on checkpoints from 30 to 50 epochs ?

The value I get is 0.297 (EAO), and I was wondering if its reasonable. And that after I tune it with tune_tpe.py should I expect it to reach somewhere near 0.327 (EAO), as reported in the paper.

It should be close at least.