search

facebookresearch / co-tracker

CoTracker is a model for tracking any point (pixel) on a video.
https://co-tracker.github.io/
Other
2.52k stars 177 forks source link

possible applications #3

Open lucasjinreal opened 11 months ago

lucasjinreal commented 11 months ago

this result is impressive, does there any possible downstream applications examples which this tech can be used?

nikitakaraevv commented 11 months ago

Thank you, @lucasjinreal! I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.

Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.

lucasjinreal commented 11 months ago

@nikitakaraevv thanks, does the computation heavy (for example, able to run on mobile devices like snapchat visualeffects)? Also wanna ask, Does CoTracker able to get real-world 3d tracking results which could be used in 3d body estimation (3d body estimation suffering on tracking people or moving camera) ?

nikitakaraevv commented 11 months ago

Hi @lucasjinreal! If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.

CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)

lucasjinreal commented 11 months ago

@nikitakaraevv Looks promising, so, is CoTracker able to run realtime on CPU? I think it can be more be usable if combined with appearing StableDiffusion's controlnet swap body from user's video.

nikitakaraevv commented 11 months ago

Thanks, @lucasjinreal! CoTracker is currently unable to run in real-time on a CPU as it is still not fast enough for such a use-case. And thanks for your suggestion, I will try to add examples with CoTracker applications to this repo later.

DavidTu21 commented 11 months ago

Hi @nikitakaraevv,

Thank you for your amazing work. I am working on a project aimed at segmenting and reconstructing 3D objects from head-mounted AR device scanned data previously with mask-rcnn and tsdf integration. Recently I tried SAM and Co-tracker and found them to work very well and I'm considering integrating them into my work in the future.

You've previously noted that "CoTracker could improve the 3D reconstruction of both static and dynamic objects." Could you provide some specific insights into how CoTracker achieves this improvement? I'm particularly interested in understanding the underlying mechanics and the particular aspects where the enhancement is most noticeable. Your expertise on this subject would be greatly appreciated.

Thank you for your contribution to the field and for your time in addressing this inquiry.

nikitakaraevv commented 11 months ago

Hi @DavidTu21, thank you!

Did you replace PIPs with CoTracker in this project? It would be interesting to see any numbers or results if you could share them!

We're currently working on 3D reconstruction using CoTracker, but we don't have any specific insights yet. This is just our current intuition. Stay tuned!

DavidTu21 commented 11 months ago

Hi @nikitakaraevv, thank you for your reply! I haven't replaced PIPs with CoTracker in that project, I was using a different tracking method with SAM previously which is called Track-Anything, and realised that I could also try the CoTracker for point-based tracking. I will give it a go and let you know if there are any interesting results :)

abhishekmonogram commented 8 months ago

@nikitakaraevv Can co-tracker work real time on video stream from a webcam? Are there any examples for this?

GeorgeAdamon commented 8 months ago

Hi @lucasjinreal! If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.

CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)

This is a truly amazing project! My project involves the tracking of a moving camera (filming ice-skaters), and this algorithm truly outperforms anything that is out there in terms of point tracking accuracy and consistency. This is the first step in any camera tracking solution.

Unfortunately, the second step of camera tracking, what is commonly referred as "3D Solve" seems to be outside the scope of this research. Is anyone aware of a project that combines the outputs of co-tracker with a 3D camera solver ?

nikitakaraevv commented 6 months ago

Hi @abhishekmonogram, yesterday, we released a new model that has an online mode, please see this issue.

nikitakaraevv commented 6 months ago

Hi @GeorgeAdamon, thank you! Yes, we are working on that as well. You can check out our new paper https://vggsfm.github.io/

AshkanTaghipour commented 4 months ago

Thank you, @lucasjinreal! I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.

Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.

Hi,

Thank you for your comprehensive paper; I really enjoyed reading it. I have a question regarding the application of version 2 (v2) in point tracking with the "Segment Anything" feature. I've read some of your comments about the performance drop of v2 in the previous code base. Could you please clarify if it's still feasible to use v2 for point tracking with "Segment Anything"?

Thank you!

nikitakaraevv commented 4 months ago

Hi @AshkanTaghipour, I haven't tried it myself, but it should be similar to v1, only much more efficient. As a rule of thumb, the more points you are tracking at a time, the better the performance.