Poor tracking performance

[iandreariley]

I've successfully installed your program, but the tracking performance poor. I'm pretty confident this is just an error somewhere in my configuration. Perhaps the default hyperparameters in tracker.m don't match the pretrained network I'm using?

I am using the pretrained weights in 2016-8-17.net.mat. I've also tried adding the stats file ILSVRC15.stats.mat to the configuration, but this didn't seem to affect the behavior of the tracker. I'm using matlab r2017b and I installed matconvnet from its github repo today; I presume that's the most recent version (v1.0-beta25).

Specifically, the tracker tends to drift to the left in most of the videos, which seems like a strange and non-random characteristic.

Any help here is appreciated. Thanks so much for making this work public.

[jvlmdr]

It does sound like an error. Some people have complained that the code does not work with recent versions of matconvnet. At the time that we published the code, we were using v1.0-beta20 (see README.md). I'm not certain, but perhaps this is the issue.

[iandreariley]

Ok, I'll try using v1.0-beta20 and see if there is a different. Thanks!

[iandreariley]

@jvlmdr, reverting to matconvnet v1.0-beta20 doesn't seem to change the tracking performance. However, another possible issue became apparent: I'm using cuda-8.0, which defines an addAtomic(double*, double) function. Matconvnet 1.0b20 also defines an addAtomic, so in order to compile matconvnet 1.0b20 I deferred to the cuda definition in the manner described in this SO post. I didn't see this issue in 1.0b25, presumably because by then, the matconvnet definition was removed in favor of the cuda-8.0 definition.

Have you tested this on cuda 8.0? That might be the issue ... although it does seem odd that addAtomic would return numbers different enough to affect tracking that dramatically.

In any case, my real aim is to train a smaller version of your network - perhaps if its trained using cuda-8.0 it won't matter ... any thoughts?

Thanks again.

[KengChiLiu]

@iandreariley Hi there, I'm recently studying in tracking problem too. I've installed this code on matlab 2016b, cuda8.0 and matconvnet v1.0-beta25, but seems to be fine now. However, maybe you can describe your problem more in detail.

[iandreariley]

@KengChiLiu Thanks for the information. Here's a more thorough description of the problem.

Problem: Poor tracking performance. The bounding box follows the target for a few frames at most and then loses the target completely. Notably, in the few sequences I've watched, it veers off to the left. This was a consistent enough pattern that I looked for errors in the bounding box code thinking that maybe it was getting an incorrect first frame, but I couldn't find any problems.

Setup Matlab 2017b matconvnet v1.0-beta20 and beta25 cuda-8.0 cudnn Ubuntu 14.04

NN Setup weights: 2016-8-17.net.mat (downloaded from the project page's "color" link.) stats: ILSVRC15.stats.mat (downloaded from the project page) configuration (hyperparameters): default

Matconvnet compiled with following

vl_compilenn('enableGpu', true, ...
               'cudaRoot', '/Path/to/CUDA-8.0', ...
               'cudaMethod', 'nvcc', ...
               'enableCudnn', true, ...
               'cudnnRoot', '/Path/to/cudnn') ;

note: vl_testnn('gpu', true) runs without any failures.

Thoughts: Possibly one or several of the hyperparameters don't match the network weights (i.e. it was trained with other hyperparameters) or the stats don't match the weights. It may also be some difference in matlab 2017b, although that strikes me as unlikely - there are no errors in the code, just poor performance.

[KengChiLiu]

@iandreariley Is the result of the demo sequence normal??? If it is, I don't think the problem you met is caused by versions. In addition, maybe you can run some example that are the same as the authors put on their project page to see whether the results meet.

[abcilike]

Good in matlab 2017a, bad in matlab 2017b. I have tried.

[amoudgl]

I also observed the similar behaviour as pointed by @abcilike. After the first frame, the tracker was drifting fast in Matlab R2017b. I tried the same exact code in Matlab R2016a and it worked perfectly.

[stomachacheGE]

I worked on Matlab R2018a and observed similar behaviour as in @abcilike and @amoudgl 's cases. Will try with Matlab R2016.

[TSSSS4]

Works well on Matlab 2017a，but get worse on Matlab 2018a

[guwenning]

@KengChiLiu Hello, I don't quite understand the 8 coordinates in ground_truth. I used a trace with a number of coordinates of 4. I found that the coordinates of the first frame did not completely wrap the target on the target. If I use my own sequence of images, I should enter the first What should be the coordinate format of a frame of picture?

[lgdhang]

I observed similar behaviour with Matlab R2018a. I found there was a numerical difference between the results of imresize on the CPU and GPU. Computing imresize function on CPU can solve the problem.

[JellyWong]

I got the same observations as @lgdhang. I have tested the Matlab R2018b and R2017b on windows10 with matconvnet v1.0-beta20 and Cuda-8.0. The performance is quite horrible and the target tends to drift to the left. However, when I moved the ubuntu16.04 (Cuda-9.0, Matlab R2016b) with the same set of code, the tracker performed quite well. After line-by-line debugging, I found that the inconsistent performance is caused by the numerical difference between the implementations of imresize(GPU) and imresize(CPU). You're expected to perform imresize in CPU and then migrated back to GPU to align the performance reported in the paper.

Specifically, if you just do testing, you can simply change the following line

https://github.com/bertinetto/siamese-fc/blob/f586d4e32091184240ab7b496ea4df9cb81dd001/tracking/tracker_eval.m#L21

to

responseMapsUP(:,:,s) = gpuArray(imresize(gather(responseMaps(:,:,s)), p.responseUp, 'bicubic'));

[CXSunshine]

I got the same observations as @lgdhang. I have tested the Matlab R2018b and R2017b on windows10 with matconvnet v1.0-beta20 and Cuda-8.0. The performance is quite horrible and the target tends to drift to the left. However, when I moved the ubuntu16.04 (Cuda-9.0, Matlab R2016b) with the same set of code, the tracker performed quite well. After line-by-line debugging, I found that the inconsistent performance is caused by the numerical difference between the implementations of imresize(GPU) and imresize(CPU). You're expected to perform imresize in CPU and then migrated back to GPU to align the performance reported in the paper. Specifically, if you just do testing, you can simply change the following line

  siamese-fc/tracking/tracker_eval.m

     Line 21
  in
  f586d4e

       responseMapsUP(:,:,s) = imresize(responseMaps(:,:,s), p.responseUp, 'bicubic'); 

to responseMapsUP(:,:,s) = gpuArray(imresize(gather(responseMaps(:,:,s)), p.responseUp, 'bicubic'));

tracking：what is the output of the tracking？

[Zhanglei-zl]

I got the same observations as @lgdhang. I have tested the Matlab R2018b and R2017b on windows10 with matconvnet v1.0-beta20 and Cuda-8.0. The performance is quite horrible and the target tends to drift to the left. However, when I moved the ubuntu16.04 (Cuda-9.0, Matlab R2016b) with the same set of code, the tracker performed quite well. After line-by-line debugging, I found that the inconsistent performance is caused by the numerical difference between the implementations of imresize(GPU) and imresize(CPU). You're expected to perform imresize in CPU and then migrated back to GPU to align the performance reported in the paper.

Specifically, if you just do testing, you can simply change the following line

https://github.com/bertinetto/siamese-fc/blob/f586d4e32091184240ab7b496ea4df9cb81dd001/tracking/tracker_eval.m#L21

to

responseMapsUP(:,:,s) = gpuArray(imresize(gather(responseMaps(:,:,s)), p.responseUp, 'bicubic'));

Thank you, After change the code I have solved this problem!