search

facebookresearch / co-tracker

CoTracker is a model for tracking any point (pixel) on a video.
https://co-tracker.github.io/
Other
2.52k stars 175 forks source link

question about v2 #54

Open XiaoyuShi97 opened 6 months ago

XiaoyuShi97 commented 6 months ago

Hi, I find that you mention the model is re-trained. I wonder if the old checkpoint is still compatible with v2 implementation? thanks！

nikitakaraevv commented 6 months ago

Hi @XiaoyuShi97! Theoretically, the old checkpoint can work with the new codebase, but it is probably going to be worse than the new checkpoint because some components are implemented slightly differently. However, you can still use the old version. Please see https://github.com/facebookresearch/co-tracker?tab=readme-ov-file#previous-version

XiaoyuShi97 commented 6 months ago

Thanks for your prompt reply. In my case, another project uses frozen cotracker v1 as a sub-module. But when I substitute it with v2, the performance drops a lot. Is there possible reason for this drop? Or can I re-write v1 following your v2 implementation so that I can continue to use v1 with batch size larger than 1? Thanks!

nikitakaraevv commented 6 months ago

Hmm, could you tell me what exactly is happening? Maybe you can send a video with predicted tracks if possible? Yes, it should be relatively easy to use the old checkpoint with the new codebase. You'd need to use the old UpdateFormer without the virtual tracks.

HeadReaper-hc commented 6 months ago

Hi, may I ask if v2 requires more GPU memory or less than v1? @XiaoyuShi97 @nikitakaraevv

nikitakaraevv commented 6 months ago

Hi @HeadReaper-hc, when tracking more than ~100 points, v2 requires much less GPU memory for the same number of points

HeadReaper-hc commented 6 months ago

@nikitakaraevv get it. Thank you very much.

1242433760 commented 5 months ago

@nikitakaraevv Hi, another question, does v1 support online mode ?

nikitakaraevv commented 5 months ago

Hi @1242433760, only if you manage to load the v1 checkpoint in the new codebase.

Hi @XiaoyuShi97! Theoretically, the old checkpoint can work with the new codebase, but it is probably going to be worse than the new checkpoint because some components are implemented slightly differently. However, you can still use the old version. Please see https://github.com/facebookresearch/co-tracker?tab=readme-ov-file#previous-version