video stream and audio stream out of sync

[kocoten1992]

Have you read the FAQ and checked for duplicate open issues? Yes, I search with keyword sync, didn't find related issue

What version of Shaka Player are you using? v3.0.2

Are you using the demo app or your own custom app? Custom app

What browser and OS are you using? GoogleChrome, Ubuntu

Sometimes, when watching long movie, video and audio (using DASH) going out of sync, just half a second or a second (but I can realize that - because character lip movement), fix it by click on the seeking bar, audio and video will sync itself.

I've experience this a number times, don't know how to reliably reproduce them.

On the otherhand, youtube seem to have this problem as well, but they cleverly fix that by either pausing video or speed up the video until it match. (it happen really fast, like in 0.2s - anyone else experience this ?)

Just want to ask is it possible for our library to out of sync video and audio? And we currently don't have mechanism to periodic autosync, is that right ?

[joeyparrish]

Audio-video sync is the responsibility of the browser. The player feeds segments to the browser, and those segments contain timestamps that are used to sync video frames to the audio. The player can add offsets to the segments to change sync, but we only do this with values indicated by the DASH manifest.

In your case, since seeking seems to fix the problem, this would point to the browser's rendering being the problem. Seeking does not in any way change the offsets that might be applied to segments by the player.

I'd like to ask some follow-up questions, though, to help you find the best answer. I may end up asking you to file a chromium bug report if it starts to look like that might the cause.

• Does this happen consistently? If you play the same video for the same length of time, do you see the sync problem at around the same time?
• How powerful is your Linux system? How many cores do you have, at what speed? Is your system heavily loaded at the time you're watching the video?
• What's the lowest resolution available for this content? If you disable ABR and select that lowest resolution, does the problem go away?
• Was the Shaka Player tab in the background at any time during your test, or was it always the "active" tab in its Chrome window?
• What software, service, or appliance was used to encode the DASH content?

Thanks!

[kocoten1992]

Hi,

• It happen consistently, about 10-15 minutes in (doesn't matter where I start), I see syncing problem (about 0.1 or 0.2 at that point, but easily recognizable).
• It an old one, https://www.cpubenchmark.net/cpu.php?cpu=AMD+Athlon+X4+860K&id=2362, system is idle (fullscreen movie watching on browser).
• Lowest is 240p, it is still out of sync after a while (I test in 15 mins), cpu usage is low.
• Watching fullscreen, the tab is always active (I make sure that by move mouse a little periodicly).
• software is FFmpeg, video using this guid: https://developers.google.com/media/vp9/settings/vod, audio: opus 96k, MPD using: http://wiki.webmproject.org/adaptive-streaming/instructions-to-playback-adaptive-webm-using-dash

I suspect if I do a rogue seek (seek to current time once in a while, this problem would be gone), I will do an experiment and tell you.

(I could send you all the stream and MPD if you interest).

Thanks.

[joeyparrish]

I would be interested to have the original video and try encoding it with our tools and the version of ffmpeg and libvpx that we have. If it's something you can share, even privately to shaka-player-issues@google.com, I would appreciate that. If I can re-encode it and the problem goes away, then we know the issue is in the content encoding.

I would also be interested to know if you see this issue with any long-running content that is built into our demo app. Sintel is only 14 minutes long and has very little dialog, so that might be a poor choice. But I'm not sure there's anything longer there. Tears of Steel is 12 minutes long and has plenty of dialog. The last line ends at about 9:25. I'm not sure if that's enough time, but try this for us: https://shaka-player-demo.appspot.com/demo/#asset=https://storage.googleapis.com/shaka-demo-assets/tos-ttml/dash.mpd

I can also produce a copy of Tears of Steel that loops several times to make it long enough to trigger this issue. I can even restrict it to VP9 and Opus to more closely match your content setup. I'll work on this tomorrow (Thursday, US Pacific time) and send you a link.

It would also be informative if you could try reproducing your content with H264 and AAC. If it is codec-specific, that would be very signficant.

These things would all help us determine the difference between a content encoding issue and a browser bug.

[kocoten1992]

Hello, I did a number of experiment and want to point out a few thing before closing this issue:

1. So the re-encode content does indeed being error (for some reason ffmpeg encode 1080p only about 60% and 720p 95% of the real movie - the filesize indicate that, using ffprobe DURATION further confirmed that re-encode content is bug), but that only happen on 1080p and 720p (other resolution like 480p, 360p and 240p have correct duration - and yet still out of sync).
2. I'm also testing on firefox, the re-encode content doesn't get out of sync even on bugged stream (kudo to them :+1: )
3. I try different movie, when encoding go smoothly - every stream correctly encoded, out of sync phenomenon don't happen.

I don't know what direction to take from here. So yes, that could indicate chromnium bug on my bug content. I guess after encode, I need to confirm duration with the original duration.

Think I'm gonna close this (I don't know if chrominum team will agree fix bug on bug content), if you think otherwise, please open and I'll try.

[joeyparrish]

Thanks for the information!

Based on my experience with content issues in general, I would say this is bad content. Even if Firefox plays it, that doesn't mean it's necessarily in-spec. Some players or platforms are more forgiving or flexible about their interpretations of things.

My suggestion is to check if you're using a recent, stable release version of ffmpeg. For example, I see good results with v4.3.1, which I installed from a Debian package. That appears to be the latest release as of August 2020. Before that, I was running v4.2.1 built from source.

Then I would suggest you check that you have a recent stable version of libvpx. I have seen many, varied issues with VP9 content encoded with old versions of libvpx. I'm currently using libvpx v1.8.2 from a Debian package. There's a newer release out as well, v1.9.0. I have created bad VP9 content before that was fixed by upgrading libvpx, even without upgrading ffmpeg. So I would suggest upgrading if your copy is older than v1.8.2.

With these versions, our own demo and test content seems to be in sync. We encode everything in both H264/AAC and VP9/Opus in the same MPD (using Shaka Streamer).

I hope that helps!

[kocoten1992]

Hi @joeyparrish ,

I'm currently re-encoding some movie in vp9 1.8.2 (or 1.9.0) and libopus 1.3.1 (I actually use this one https://johnvansickle.com/ffmpeg/git-readme.txt), will let you know the result.

The thing is, I highly suspect it on chrome part:

• https://support.google.com/youtube/thread/13160885?hl=en
• https://support.google.com/chrome/thread/18668415?hl=en (it mention 10 min, I feel the same, also if video run nonstop for about 1h, sync could get on second level - I measure 3-4s for an hour).
• https://support.google.com/chrome/thread/5356407?hl=en (this one mention firefox doesn't get desync)

If this issue still persist, may I ask for your help on this :) ?

[kocoten1992]

I'm happy to confirm, after reencode using https://johnvansickle.com/ffmpeg/:

• audio: libopus 1.3.1
• video: libvpx 1.9.0-61-gc413c8f18

Audio and Video still having syncing problem on google chrome, this is huge :P, cc @joeyparrish

[kocoten1992]

Hi,

This issue is driving me crazy.

I've some older video already encoded (also using ffmpeg from johnvansickle, vp9 and opus), at that time I didn't take note which version of libvpx and libopus was. Let call this movie C (correctly sync). C did not show any sight of mis-sync (tested with latest chrome - Version 85.0.4183.83 (Official Build) (64-bit) ). That would explain why I see this:

3. I try different movie, when encoding go smoothly - every stream correctly encoded, out of sync phenomenon don't happen.

The movie I reencode (let call it M - mis-sync) was previously wrongly sync, and after reencode with latest codec, it still mis-sync

I'm happy to confirm, after reencode using https://johnvansickle.com/ffmpeg/:

Previously, I was mistakenly think that maybe C could be cause by libvpx or libopus, but that wasn't the case, honestly I don't know what could possibly cause them - it could be the source content itself.

I not sure you could replicated it with simply loop Tears of Steel, I think I need to send you the actual source content that I'm using, but when I'm watching source using VLC or MPV media player, I didn't get mis-sync. After encode to mpd, watching on Firefox also didn't get mis-sync. It could be source - but why other are playing just fine ? Then Chrome is disagreeing with many major media player, and need to be fixed.

I'll send my source content and encoded content + mpd to you via shaka-player-issues@google.com.

Thanks!!

[kocoten1992]

Fixed it!

Been doing research extensively, if you are incharge of transcoding process, you could fix this, even if content is bad, take a look at these: http://dranger.com/ffmpeg/tutorial05.html

In layman term, there are two keywords to care about:

1. aresample=async=1000 (https://ffmpeg.org/ffmpeg-all.html#Examples-50)
2. use_wallclock_as_timestamps

(I don't know about 2, use 1 already fixed my problem)

Hope this will help somebody.

[joeyparrish]

@kocoten1992, thank you for posting the solution! I'm sure this will benefit others in the community having AV sync issues.