search

facebookresearch / co-tracker

CoTracker is a model for tracking any point (pixel) on a video.
https://co-tracker.github.io/
Other
3.93k stars 253 forks source link

cannot backward tracking in online model with specific queries in middle frame #60

Open fangli333 opened 10 months ago

fangli333 commented 10 months ago

Hi,

I am using your online model for point tracking. and I want to process a video. I want to set up specific queries in one of the middle frame and try to use backward tracking. here is my code:

`video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device) cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2_online").to(device) cotracker(video_chunk = video, is_first_step = True, queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True) vis = Visualizer(save_dir="./co-tracker/result", pad_value=120, linewidth=2) for ind in range(0, video.shape[1] - cotracker.step, cotracker.step): pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True)

vis.visualize(video, pred_tracks, pred_visibility)`

and here is the bug i got: ---> 60 pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True) 61 62 vis.visualize(video, pred_tracks, pred_visibility)

~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.9/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, kwargs) 25 def decorate_context(*args, *kwargs): 26 with self.clone(): ---> 27 return func(args, kwargs) 28 return cast(F, decorate_context) 29

TypeError: forward() got an unexpected keyword argument 'backward_tracking'

nikitakaraevv commented 10 months ago

Hi @fangli333, thank you for the question! Backward tracking is only supported in the offline mode because you wouldn't need an online algorithm if you already have access to all the video frames.

Davidyao99 commented 6 months ago

@nikitakaraevv I think this might be useful for those looking to run cotracker on long videos and trying to avoid OOM issues. I implemented a backward_tracking on my own with online cotracker to resolve this. If you think it might be a useful feature, I can clean and pull_request my code.

nikitakaraevv commented 5 months ago

Hi @Davidyao99, thank you! Could you open a PR?

Davidyao99 commented 5 months ago

Hello @nikitakaraevv, sure, I submitted a PR here. I modified online_demo.py to perform backward tracking, and also take in segmentation masks. Let me know if you rather it be in a separate module / file. Thanks!