search

facebookresearch / co-tracker

CoTracker is a model for tracking any point (pixel) on a video.
https://co-tracker.github.io/
Other
2.52k stars 175 forks source link

question about backward_tracking #66

Open XiaoyuShi97 opened 5 months ago

XiaoyuShi97 commented 5 months ago

https://github.com/16lemoing/dot/blob/e32c6f7de12342460371f2efc4789bd79c4a39a3/dot/models/shelf/cotracker2_utils/predictor.py#L145-L153

Hi, I am a bit confused about backward_tracking. First, back_tracking means computing the trajectory starting from the last frame to the first N-1 frames? If so, does backward_tracking benefit (get higher accuracy) from forward_tracking? And do the variables "tracks and visibilities" in L153 overwrite those in L145? Thanks!

nikitakaraevv commented 4 months ago

Hi @XiaoyuShi97, CoTracker is an online algorithm, and can only track forward. That's why there's such a thing as backward tracking, where we simply run tracking forward on a reversed video.

The variables "tracks and visibilities" in L153 overwrite those in L145 only for the frames for which the model didn't predict any tracks. For example, if a point was queried at a frame number 10, forward tracking will predict tracks for frames 10,11,12,..., while backward tracking will add predictions for 9,8,7,...