search

Epiphqny / VisTR

[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers
https://arxiv.org/abs/2011.14503
Apache License 2.0
739 stars 95 forks source link

How to process the video that has more than 36 frames? #3

Closed jxiangli closed 3 years ago

jxiangli commented 3 years ago

Thanks for your excellent work. I have one qustion about the inference process. If the length of video is larger than 36, how to link the tracks from diffenrent clips? looking forward to your reply.

Epiphqny commented 3 years ago

@jxiangli If the video is larger than 36, you could split the video into several clips, and link them based on the result of the overlapping frame using traditional post-processing.

KainingYing commented 3 years ago

@jxiangli If the video is larger than 36, you could split the video into several clips, and link them based on the result of the overlapping frame using traditional post-processing. Thanks. By the way, the max length of Youtube VIS 2021 is 72 and 84 in train and val set respectively while the 2019 is 36.

jxiangli commented 3 years ago

Thanks for your reply~ I just started to research the VIS. Your means is that using post-processing like Kalman Filter and Hungarian algorithm to link the adjacent tracks, am I right?

Epiphqny commented 3 years ago

Thanks for your reply~ I just started to research the VIS. Your means is that using post-processing like Kalman Filter and Hungarian algorithm to link the adjacent tracks, am I right?

@jxiangli That's right.

jxiangli commented 3 years ago

Thank you very much!

Epiphqny commented 3 years ago

@jxiangli @noobying Thanks for pointing out it, I have not tried Youtube VIS 2021 before. In the original setting of VisTR, the number of instance queries is proportional to the number of input frames, and the instance queries are a fixed number of learned embedding. Therefore the number of input frames is fixed. But in row 3 of Table 1(b) of the paper, we also experiment with the instance-level queries that do not rely on the number of frames. In this way, the model could process a dynamic number of frames. But two many frames could also raise a memory issue, we leave it as future work.