search

facebookresearch / co-tracker

CoTracker is a model for tracking any point (pixel) on a video.
https://co-tracker.github.io/
Other
3.84k stars 251 forks source link

Tracking original coordinate over chunks (Real-time) #125

Open sarthakg2002 opened 3 weeks ago

sarthakg2002 commented 3 weeks ago

I want to use the model for real-time video feed. I have introduced a start delay to fill up a buffer and then make the video playback smooth using chunking. However when switching between chunks, if there is an object over that coordinate, the subsequent chunks start tracking the object instead of the original coordinate. Is it possible to maybe only track the original point across the chunked video, maybe don't change the original information when tracking while sending new frames?

bbmedia commented 1 week ago

@sarthakg2002 I try also to track in real-time to compute the homogenous transformation matrix etc... It isn't very stable in my case and not too fast (3-4fps with RTX 3080 on a 848 x 480 feed).

Can you maybe show me, how you did integration of cotracker in your case?

sarthakg2002 commented 1 week ago

@bbmedia I used a few strategies to speed it up but still i could only get it around 30 fps (for context a 30 second input video took around 58 seconds). I used multithreading with overlap buffer chunks. I also added a start delay such that the buffer could fill up before starting displaying. So when one of the buffers was full and being displayed, another buffer will fill up and i would switch between the buffers. I used a 50% overlap between the chunks. But there is a problem with tracking. When switching between chunks, if the tracked coordinate is over another object, the subsequent frames start tracking it.