More Types of Video Annotations?

[SiftingSands]

Does Label Studio support video annotations with other shapes besides bounding boxes, such as polygon or pixel-wise masks, spatial keypoints, text overlay on a shape (like https://labelstud.io/templates/optical_character_recognition.html for videos)? The "control tags" for these features that work for images don't seem to work for videos.

The original poster in https://github.com/heartexlabs/label-studio/issues/3693#issue-1583979964 mentioned some of these features in their post about timeline segmentation, but no further detail was given more advanced spatial/geometric annotations.

Thanks!

[SiftingSands]

I also saw a more in-depth "Details" panel in one of your labeling videos (https://www.youtube.com/watch?v=pHHXdIjUBmg) than what I currently see in LabelStudio with the "Video Object Detection and Tracking" template. vs

That YouTube video is a year old now, so I wanted to know if those additional features, like dynamic metadata, is incorporated in the current release and I'm missing something. Or if this difference is intentional?

[makseq]

@SiftingSands At this moment LS doesn't support anything other than bboxes. Unfortunately we freezed the development of dynamic metadata, because we did not see the required level of demand.

[SiftingSands]

Thanks for confirming the feature set. Unfortunate to hear about the dynamic metadata being frozen. I was hoping to put in detection & ID scores, orientation (yaw, pitch, roll), activity state, etc. I'm just testing out LS for visualizing an already labeled video dataset and the timeline/tracking feature was attractive.

I'm okay with closing this issue, but I'll leave it up to you in case you want it left open.

[makseq]

I prefer to keep it opened, because it's a feature request and it will help us to get some statistics about dynamic metadata for videos. Thank you for your request :-)

[eliabruni]

I also confirm interest in this!

[shortcipher3]

I'm also interested in this feature. Basically I want to be able to draw a keep-out-zone as a polygon onto the video as a pre-annotation, the annotator then would watch the video and classify whether the keep-out-zone was ever entered. Seems like it wouldn't be much work to support this.

[hoangphucITJP]

I want polygons on videos too

[Buckler89]

Me too

[jneeven]

Segmentation mask support for videos would be great, especially in combination with SAM2 integration. I find bounding boxes not to be very useful in practice at all. Currently I extract keyframes outside of Label Studio and label them as images, and then feed them to the SAM2 propagator. But this isn't interactive on an inter-frame level and isn't flexible in the selection of keyframes, so would be awesome to have more general label support for videos in the Label Studio interface. Would it be a lot of work to support e.g. keypoints, brushtstrokes etc?