Audio has a 30-40 second lag when coming from different source from video

[pietrorea]

I'm sending H264 video and AAC audio buffers to jMuxer on the browser. The video and audio come from different "live" sources. When I do this, the video starts playing right away but audio takes at least 30 seconds to start playing. I'm not setting "duration" on the jMuxer instance, so it's inferring it from the buffers.

Separately, the Media tab in Chrome Developer tools says that some timestamps are out of sync:

"Failed to reconcile encoded audio times with decoded output.".

This is the place in the Chromium code where that gets logged: https://chromium.googlesource.com/experimental/chromium/src/+/refs/wip/bajones/webvr/media/filters/audio_timestamp_validator.cc

I would expect the audio to start playing right away once the audio buffer is big enough. Any tips debugging this would be appreciated.

[samirkumardas]

I'm not setting "duration" on the jMuxer instance, so it's inferring it from the buffers. Unfortunately, it is not taking timing information from the buffer. If no timing information is provided, jMuxer runs the video at 30/fps

If it gets correct audio data, it should play immediately. jMuxer has nothing to do with it, the browser plays it.

You can capture some audio data in a file and try playing using a player e.g VLC player to see it plays immediately.

Don't you feed both audio and video chunk at the same time?

[pietrorea]

@samirkumardas thank you for the info.

You can capture some audio data in a file and try playing using a player e.g VLC player to see it plays immediately.

I can capture audio and save it into a file from the server and it plays immediately.

Don't you feed both audio and video chunk at the same time?

Yes, I'm using the same feed method to add chunks to the video buffer and the audio buffer.

I also added the exact error message I see in Chrome media tools and the place in Chromium where it gets logged. My best guess is that the video buffer and the audio buffer I'm sending to the feed method have different durations.

[samirkumardas]

My best guess is that the video buffer and the audio buffer I'm sending to the feed method have different durations.

Video duration can be controlled from jMuxer with FPS or chunk duration. As you are not providing any timing information while you feed buffer or instantiate jMuxer, so the video will play at 30/fps as I mentioned earlier. However, we can not control the audio duration from jMuxer and it runs according to the sample rate.

So you must always provide video and audio of the same duration, otherwise, it will be out of sync eventually. Though, it can not be the reason for the starting delay of the audio.

You can do one thing, you can simply provide the audio feed and set jMuxer mode to audio-only to see if it still has a delay at the starting.