cannot reproduce your pre-trained model

[gongbudaizhe]

Hi, @bertinetto

I have rerun your training code for 50 epochs without any modification, but I cannot reproduce your released model for color image. Specifically, it performs worse in the OTB50 benchmark compared to your released color model in all aspects BC, DEF, IV, OCC, etc.

Could you kindly point out in which way I may have done wrong?

Attached the training metrics. net-train.pdf

[bertinetto]

Hi, So did you try your model against the pre-trained network (which one?) trained by us? There are many things that could be different if you have recreated the dataset and trained from scratch unfortunately. Here few things I can think of, but clearly everything counts:

• did you save jpeg at a decent quality? (i think i have used 90, imagemagick default is lower)
• are you using our same cudnn, matconvnet and cuda versions?
• are you setting batchnorm in test mode during tracking?
• what do you mean with "color model"? As stated in the paper, for VOT we use a network trained only on RGB image, while for OTB we use a network where 25% of the training images are randomly set to grayscale. Just to reflect the dataset composition. Do you do the same?

[gongbudaizhe]

Thanks for your consideration.

I have trained the model following procedures listed in the training and tracking section without modifying any of your released code or configurations. The pre-trained model is downloaded from your project page which is trained solely by RGB images.

• The ILSVRC15 curation code does use 90.
• I am using cuda 7.5, cudnn v5.1 and matconvnet v1.0-beta20. Did you use the same cuda and cudnn? Sorry I can't find your environment specifications about cuda and cudnn. Matlab version differs though, R2016a is used instead.
• Both models use the same tracking code, Maybe this is not what causes the performance differences?And yes, I am sure batchnorm parameters are fixed during tracking.
• By color model, I mean the model trained only on RGB images. I want to reproduce your color model so I didn't use any gray images. I test both models(pre-trained model and model trained from scratch) in the OTB benchmark just to check they perform about the same.

I will try using Matlab R2015b and see whether it will make any difference. : )

[bertinetto]

• Actually we have used cuda 7.5 and cudnn v5 at the time.
• As said in the paper, we trained the network with 25% chance grayscale, to reflect the composition of the dataset.

[gongbudaizhe]

Thanks for your prompt reply.

• Training has just started using exactly the same library configurations(cuda 7.5, cudnn5.0, matlab 2015b, matconvnet v1.0-beta20). I will post the results as soon as they are available.
• I get your point : ) Sorry I wan't clear enough. I am not trying to reproduce your tracking results at the OTB benchmark. In that case, I should use 25% chance grayscale to train the color+gray model as stated in the paper. Instead, I want to reproduce the color model(model trained sole on RGB images), and by using OTB benchmark or any other benchmark, I didn't want to get the best possible performance but simply as a way of measuring the differences between pre-trained color model and the model trained from scratch using codes and configurations released in this repository.

[gongbudaizhe]

Hi,

The training has finished. I get similar models now using matlab R2015, cuda 7.5, cudnn 5.0 and matconvnet v1.0-beta20.

Thanks!

[bertinetto]

That's awesome! Thanks for signalling the issue and taking the time to retrain!

On 9 Jan 2017 12:27, "Bi Li" notifications@github.com wrote:

Hi,

The training has finished. I get similar models now using matlab R2015, cuda 7.5, cudnn 5.0 and matconvnet v1.0-beta20.

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bertinetto/siamese-fc/issues/13#issuecomment-271262433, or mute the thread https://github.com/notifications/unsubscribe-auth/ADLRue75MHLD_Pj6EVS9Xmb5PlZlT_Daks5rQhmQgaJpZM4LZEU3 .