Synchronized video playback for multiple videos/channels and possibly a split view for video object detection and tracking

[daniel1896]

Is your feature request related to a problem? Please describe. I would like to be able to label videos with multiple channels (e.g. RGB and thermal) for video object detection and tracking in Label Studio. However, it is currently difficult to label these videos accurately and efficiently because I cannot view both channels simultaneously or synchronize their playback.

Describe the solution you'd like I would like to see a new feature in Label Studio that allows me to label videos with multiple channels more accurately and efficiently. Specifically, I would like to see a single video player that displays both channels side-by-side, or even better, with a split line that I can drag to reveal more of one channel or the other.

With this feature, I could label both channels simultaneously and synchronize their playback within the same interface. This would improve the accuracy and efficiency of my labeling process and save me time and effort.

Describe alternatives you've considered Currently, there is no way to view multiple video channels simultaneously in Label Studio, so I am not aware of any alternatives that would allow me to label these videos more accurately and efficiently.

Additional context Here are some additional details that may be helpful:

• I am labeling videos with two channels: RGB and thermal
• I would like to be able to see both channels side-by-side within the Label Studio interface, with the ability to adjust the split line between them
• This feature would improve the accuracy and efficiency of my labeling process and would save me time and effort

Thank you for considering this feature request!

[makseq]

Thank you for your detailed feature request! It's great to hear about your specific use case and the need for multi-channel video labeling. You can check this labeling config with sync functionality across different video sources:

<View style="display: flex;">

    <View style="width: 50%; margin-right: 2em;">
        <Header value="Front"/>
        <Video name="videoF" value="$videoF" sync="videoF"/>

        <VideoRectangle name="boxF" toName="videoF"/>
        <Labels name="vlF" toName="videoF">
            <Label value="Human" background="#1BB500"/>
            <Label value="Car" background="#FFA91D"/>
            <Label value="Truck" background="#358EF3"/>
            <Label value="Motorcycle" background="#FFA39E"/>
        </Labels>

        <Header value="Left"/>
        <Video name="videoFL" value="$videoFL" sync="videoF"/>

        <VideoRectangle name="boxFL" toName="videoFL"/>
        <Labels name="vlFL" toName="videoFL">
            <Label value="Human" background="#1BB500"/>
            <Label value="Car" background="#FFA91D"/>
            <Label value="Truck" background="#358EF3"/>
            <Label value="Motorcycle" background="#FFA39E"/>
        </Labels>

    </View>

    <View style="width: 50%; margin-right: 2em;">
        <Header value="Rear"/>
        <Video name="videoR" value="$videoR" sync="videoF"/>

        <VideoRectangle name="boxR" toName="videoR"/>
        <Labels name="vlR" toName="videoR">
            <Label value="Human" background="#1BB500"/>
            <Label value="Car" background="#FFA91D"/>
            <Label value="Truck" background="#358EF3"/>
            <Label value="Motorcycle" background="#FFA39E"/>
        </Labels>

        <Header value="Right"/>
        <Video name="videoFR" value="$videoFR" sync="videoF"/>

        <VideoRectangle name="boxFR" toName="videoFR"/>
        <Labels name="vlFR" toName="videoFR">
            <Label value="Human" background="#1BB500"/>
            <Label value="Car" background="#FFA91D"/>
            <Label value="Truck" background="#358EF3"/>
            <Label value="Motorcycle" background="#FFA39E"/>
        </Labels>

    </View>
</View>

[daniel1896]

Thank you for suggesting this feature of Label Studio. This would indeed work for the task, that I am trying to accomplish. However, as you can see in the screen recording below, the playback of the videos is not synchronized.

https://user-images.githubusercontent.com/29599302/235854310-77834105-9ae7-4e77-b956-5b8ab0d6008e.mp4

(Please don't mind the overexposed recording.)

[chris24sahadeo]

@daniel1896 I am also interested in this config. Were you able to get this working?

CC @makseq - looks like Daniel had some trouble with the video sync.