search

votchallenge / toolkit

The official VOT Challenge evaluation and analysis toolkit
http://www.votchallenge.net/
GNU General Public License v3.0
158 stars 46 forks source link

realtime experiments - penalize tracker if frame processing takes longer than one frame duration #15

Open daniel-d-koenig opened 4 years ago

daniel-d-koenig commented 4 years ago

To my mind, the trackers should be penalized if they need longer than one frame duration. In the current implementation it is tolerated that the processing of one frame can take longer than one frame duration as long as not more than twice the frame duration. Is this correct (see code line below)?

https://github.com/votchallenge/vot-toolkit-python/blob/b23463652bfaba2393079b3517c4bc8bae3e1de8/vot/tracker/__init__.py#L250

Maybe the if statement "self._time > self._interval:" should be changed to "self._time > 0:", or is the tolerance for over one frame desired? I did not found any information about that.

Thanks in advance.

lukacu commented 4 years ago

Real-time stuff is very tricky to measure. There are so many things that have to be taken into account. As far as I have tested the configuration it is approximately right, we created some dummy trackers with a fixed sleep interval and see what are the results.

The main focus of the experiment at the moment is to provide some time-related penalties, but we will also consider your remark in the future version of the code.

lukacu commented 4 years ago

If this is still relevant: can you provide an example with a mockup tracker that would demonstrate the problem?

daniel-d-koenig commented 4 years ago

I have no working example, I just detected this by reading and trying to understand the code. To my mind, following code line https://github.com/votchallenge/vot-toolkit-python/blob/b23463652bfaba2393079b3517c4bc8bae3e1de8/vot/tracker/__init__.py#L250 should be changed to if self._time > 0: this leads to a penalization of the tracker if the processing of the frame takes longer than one frame.