cannot backward tracking in online model with specific queries in middle frame

[fangli333]

Hi,

I am using your online model for point tracking. and I want to process a video. I want to set up specific queries in one of the middle frame and try to use backward tracking. here is my code:

`video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device) cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2_online").to(device) cotracker(video_chunk = video, is_first_step = True, queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True) vis = Visualizer(save_dir="./co-tracker/result", pad_value=120, linewidth=2) for ind in range(0, video.shape[1] - cotracker.step, cotracker.step): pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True)

vis.visualize(video, pred_tracks, pred_visibility)`

and here is the bug i got: ---> 60 pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True) 61 62 vis.visualize(video, pred_tracks, pred_visibility)

~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.9/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, kwargs) 25 def decorate_context(*args, *kwargs): 26 with self.clone(): ---> 27 return func(args, kwargs) 28 return cast(F, decorate_context) 29

TypeError: forward() got an unexpected keyword argument 'backward_tracking'

[nikitakaraevv]

Hi @fangli333, thank you for the question! Backward tracking is only supported in the offline mode because you wouldn't need an online algorithm if you already have access to all the video frames.

[Davidyao99]

@nikitakaraevv I think this might be useful for those looking to run cotracker on long videos and trying to avoid OOM issues. I implemented a backward_tracking on my own with online cotracker to resolve this. If you think it might be a useful feature, I can clean and pull_request my code.

[nikitakaraevv]

Hi @Davidyao99, thank you! Could you open a PR?

[Davidyao99]

Hello @nikitakaraevv, sure, I submitted a PR here. I modified online_demo.py to perform backward tracking, and also take in segmentation masks. Let me know if you rather it be in a separate module / file. Thanks!