search

z-x-yang / Segment-and-Track-Anything

An open-source project dedicated to tracking and segmenting any objects in videos, either automatically or interactively. The primary algorithms utilized include the Segment Anything Model (SAM) for key-frame segmentation and Associating Objects with Transformers (AOT) for efficient tracking and propagation purposes.
GNU Affero General Public License v3.0
2.77k stars 334 forks source link

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1386x768 and 256x512) while doing segtracker.add_reference() #82

Closed Coronal-Halo closed 1 year ago

Coronal-Halo commented 1 year ago

When I try to do "segtracker.add_reference(frame, label_masks)" for a second time to add new SAM mask at the second keyframe (after detecting/segmenting new objects in this frame), I got this error:

Exception has occurred: RuntimeError mat1 and mat2 shapes cannot be multiplied (1386x768 and 256x512) File "E:\Documents\Research\Segment-and-Track-Anything\aot\networks\layers\transformer.py", line 683, in fuse_key_value_id self.linear_ID_V(torch.cat([value, id_emb], dim=2)) File "E:\Documents\Research\Segment-and-Track-Anything\aot\networks\engines\deaot_engine.py", line 35, in update_short_term_memory layer_idx].fuse_key_value_id(curr_id_k, curr_id_v, curr_id_emb) File "E:\Documents\Research\Segment-and-Track-Anything\aot\networks\engines\deaot_engine.py", line 94, in add_reference_frame aot_engine.update_short_term_memory(separated_mask) File "E:\Documents\Research\Segment-and-Track-Anything\aot_tracker.py", line 69, in add_reference_frame self.engine.add_reference_frame(frame, _mask, obj_nums=obj_nums, frame_step=frame_step) File "E:\Documents\Research\Segment-and-Track-Anything\SegTracker.py", line 98, in add_reference self.tracker.add_reference_frame(frame, mask, self.curr_idx, frame_step) File "E:\Documents\Research\ImgSegGuidedMoSeg\segtracker_utils.py", line 135, in recog_seg_track segtracker.add_reference(frame, label_masks) RuntimeError: mat1 and mat2 shapes cannot be multiplied (1386x768 and 256x512)

I notice that this error only occurs when I try to call add_reference() for a second time. What's the cause of this and how can I fix it?

Coronal-Halo commented 1 year ago

Never mind, it's giving the error since I was adding the entire SAM mask instead of just the segmentation mask of new objects.