search

HumanSignal / label-studio

Label Studio is a multi-type data labeling and annotation tool with standardized output format
https://labelstud.io
Apache License 2.0
19.01k stars 2.37k forks source link

Video/audio sync doesn't work #5359

Open prielka opened 9 months ago

prielka commented 9 months ago

I am trying to do Video Timeline Segmentation in LSF, I have an issue with the audio not syncing to the video, I can play each of them separately but each play button doesn't start both video and audio together. This is the config I am using:

 <View>
            <Header>Video timeline segmentation</Header>
            <Video name="video" value="$video_url" sync="audio"/>
            <Labels name="videoLabels" toName="audio" choice="multiple">
                <Label value="Moving" />
                <Label value="Not moving" />
            </Labels>
            <Audio name="audio" value="$video_url" sync="video"/>
 </View>

I tried with several different videos, all have the same issue. I also tried different config formats, with different decoder/player combinations for the audio. Here is an example video I tried to use: https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/VolkswagenGTIReview.mp4

My LS version is 1.10.1, I am running it on Windows 11, using Chrome. This is a warning that I see in the console logs:

Waveform.js:320 The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu
value @ wavesurfer.js:5820
i @ wavesurfer.js:5718
(anonymous) @ wavesurfer.js:1846
i @ wavesurfer.js:1832
value @ wavesurfer.js:4390
value @ wavesurfer.js:4174
value @ wavesurfer.js:5582
componentDidMount @ Waveform.js:320
vl @ react-dom.production.min.js:220
Ts @ react-dom.production.min.js:259
t.unstable_runWithPriority @ scheduler.production.min.js:18
Fo @ react-dom.production.min.js:122
Ms @ react-dom.production.min.js:252
bs @ react-dom.production.min.js:243
ps @ react-dom.production.min.js:237
$s @ react-dom.production.min.js:284
(anonymous) @ react-dom.production.min.js:289
xs @ react-dom.production.min.js:244
iu @ react-dom.production.min.js:289
t.render @ react-dom.production.min.js:296
createApp @ LabelStudio.js:62
await in createApp (async)
qoe @ LabelStudio.js:31
(anonymous) @ LabelStudioComponent.js:13
commitHookEffectListMount @ react-dom.development.js:23150
commitPassiveMountOnFiber @ react-dom.development.js:24926
commitPassiveMountEffects_complete @ react-dom.development.js:24891
commitPassiveMountEffects_begin @ react-dom.development.js:24878
commitPassiveMountEffects @ react-dom.development.js:24866
flushPassiveEffectsImpl @ react-dom.development.js:27039
flushPassiveEffects @ react-dom.development.js:26984
(anonymous) @ react-dom.development.js:26769
workLoop @ scheduler.development.js:266
flushWork @ scheduler.development.js:239
performWorkUntilDeadline @ scheduler.development.js:533

I don't have any browser extensions active.

sandy1990418 commented 4 months ago

I faced the same problem and resolved it by separating the video and audio elements. Here's the my solution:

<View>
    <Video name="video" value="$video" sync="audio"/>
    <Audio name="audio" value="$audio" sync="video"/>    
    <Labels name="label" toName="audio">
      <Label value="Speech"/>
    </Labels>
    <TextArea name="transcription" toName="audio" editable="true" perRegion="true" required="true" maxSubmissions="1" rows="5" />
</View>
prielka commented 2 months ago

I tried this and it did not work unfortunately