search

z-x-yang / Segment-and-Track-Anything

An open-source project dedicated to tracking and segmenting any objects in videos, either automatically or interactively. The primary algorithms utilized include the Segment Anything Model (SAM) for key-frame segmentation and Associating Objects with Transformers (AOT) for efficient tracking and propagation purposes.
GNU Affero General Public License v3.0
2.77k stars 335 forks source link

How to deal with the disappeared objects #28

Closed ZillaRU closed 1 year ago

ZillaRU commented 1 year ago

I tried to export trackID derived by Seg-Track and got many errors. I found that there are no codes to deal with the object disappearance. https://github.com/z-x-yang/Segment-and-Track-Anything/blob/709cf2ad5ff53811f244ee19be3a8ee55290ebfb/seg_track_anything.py#L107-L117 I think Seg-Track, as a object tracking model, should handle this case better. My basic idea about this is to compare the track_mask and seg_mask like what find_new_objects does.

Will you handle the issue soon?

Thanks for your patience.

yoxu515 commented 1 year ago

Hi, once an object appear in the video, it will be assigned an unique ID and then be tracked in the whole video. Even an object disappear for a while and then re-appear, the model can still identify it by its unique ID. Could you please describe the errors more specifically?

ZillaRU commented 1 year ago

Can the values in merged pred_mask be used as track_id? I did some second-development based on ur project (using BBox as box prompt, only tracking some specific objects), maybe I missed some key operations which led to mistakes in pred_mask.

yoxu515 commented 1 year ago

Yes. The object IDs are in the pred_mask, and the mask is saved as a palette image with color indexing. You can check the visualization results. The color of the same object should be the same across different frames.

ZillaRU commented 1 year ago

I have fixed my code. Thanks for your patience.