search

kaltura / nginx-vod-module

NGINX-based MP4 Repackager
GNU Affero General Public License v3.0
1.98k stars 439 forks source link

DASH Clipping mp4 breaks AV sync #810

Open teplicko opened 6 years ago

teplicko commented 6 years ago

Hello,

I tried your module for DASH streaming with clipping input from MP4. I have vod_align_segments_to_key_frames on, because I use shaka player and it doesn't work with non-I-frame at the beginning.

I have video with I-frame every 2 seconds. When I use clipFrom, it seems like audio starts from exact clipFrom position, but video starts correctly from first I-frame position. I tried different clipFrom in 2s range and offset is exactly the time difference to the first key frame. http://127.0.0.1/dash/clipFrom/228000/,test,.mp4.urlset/manifest.mpd - perfect (even seconds are key-frames) http://127.0.0.1/dash/clipFrom/228500/,test,.mp4.urlset/manifest.mpd - 1.5s audio delay http://127.0.0.1/dash/clipFrom/229000/,test,.mp4.urlset/manifest.mpd - 1s audio delay http://127.0.0.1/dash/clipFrom/229500/,test,.mp4.urlset/manifest.mpd - 0.5s audio delay http://127.0.0.1/dash/clipFrom/230000/,test,.mp4.urlset/manifest.mpd - perfect again

I may do something wrong, because I'd suspect this kind of bug would be revealed long time ago. I checked, if there is any initial pts/dts offset, and the video doesn't have any (checked by ffprobe).

# for s in a v; do echo -n "$s: "; ffprobe -i /var/www/test.mp4 -show_frames -select_streams $s 2>/dev/null | grep pts_time | head -1; done
a: pkt_pts_time=0.000000
v: pkt_pts_time=0.040000

I'll send you examples with links (of my config, used url and test files) to your e-mail soon.

erankor commented 6 years ago

Yes, I'm familiar with this issue. We are using 2 second GOP as well, and we're making sure to send only clipFrom values which are multiples of 2 seconds. This happens because the module loads the minimum amount of data required to serve the request. So, when it gets a request for an audio segment, it doesn't even parse the video frames. And therefore, it can't align the audio according to the video frames.

teplicko commented 6 years ago

Well, will the issue be fixed soon/sometime/ever? I'd use your workaround, but I don't have all videos with the 2s GOP.

erankor commented 6 years ago

No plans to change this behavior any time soon

teplicko commented 6 years ago

Well, I'll leave the issue opened. I guess, if somebody will find the same problem, they'd like to know it's known.

mdale commented 4 years ago

Hitting this issue as well. How is this handled differently with HLS vs DASH ? We don't see the same issue with HLS.

erankor commented 4 years ago

It's because the default in HLS is to return the audio + video muxed together, while in DASH they are always separate. If you enable vod_hls_force_unmuxed_segments, HLS will probably show the same out-of-sync as DASH.

mdale commented 4 years ago

This happens because the module loads the minimum amount of data required to serve the request. So, when it gets a request for an audio segment, it doesn't even parse the video frames. And therefore, it can't align the audio according to the video frames.

Would it be possible for the module to load the video data to ensure the audio segment response is aligned based on the clipping points ? I guess you still end up with some audio samples miss-alignment relative to demuxed stream ... But would be much much better than the gap to the next keyframe.

erankor commented 4 years ago

Sorry, I'm not going to change this behavior.

zylux1 commented 3 years ago

Hi, any update on this thread? I have the same problem and it occours even in hls with vod_hls_force_unmuxed_segments enabled. Thanks