Play video and audio at same time slows down video

[borjaevo]

I have some code that plays H264. Since i added audio to the player, (in the MP4 container is in raw format, PCM_S16) video slows down, it does not go smooth. It seems that each time has to process audio packet (conversion to java audio and play) video stops for some time. I tried to paralelize code so conversion and play runs in other thread but it does no sound well. Any idea of how i have to play video and audio at same time? I thought about playing video and audio in different loops and threads. But i think it will desynchronize to easy.

[Tank-Missile]

I had the same exact issue as you. The best solution in my mind was to seperate video and audio into their own threads. I never did get the timing down though. That will require adjusting the framerate of the video. The reason the stutter occurs is because an audio packet is longer than a video packet.

[artclarke]

Creating a video player with synced audio and video is actually quite hard (regardless of the programming language or library you use), but is made even harder in Java given that you'll be somewhat the mercy of your garbage collector. Bear that in mind.

But there are a few tricks that are worth bearing in mind:

If your track has both audio and video tracks:

• Consider using separate threads for audio and video playback, and give priority to audio feedback.
• Carefully check your math (and your timebases) to make sure that you can compute how far into the audio has been consumed when you decide to render each video frame.
• DROP VIDEO FRAMES if you are behind the audio, but keep your prior frame displayed. Check your audio timing before you decode, and again before you render... basically at every point in the video thread if audio has run ahead just move to the next video frame and discard the one you're processing.

If video only, then synchronize to a system time clock rather than the audio frame.

Hope that helps.

In my non open source life, I've been able to do this with HumbleVideo (sorry can't release the code), but I also found it helpful to do the following: 1) explicitly manage the underlying frame memory (i.e. allocate and free frames). the garbage collector will eventually free humble objects for you, but you won't want to take the pause while it figures out it needs to. 2) I found using an Actor model for demuxing decoding and audio video and audio playback worked well, and I 'synchronized' audio and video based on aligning both the audio actor and the video actor to the system clock, and using pause and resume messages to the video actor if the audio actor was at risk of dropping frames. 3) I used buffering a lot. As in I would try to make sure I had audio decoded and ready to go before I started rendering / starting the video.

[dstieglitz]

We experimented with this a few years ago. Here is come code based on Xuggler which is humble's predecessor. Take a look, YMMV. It's not supported but might give you some good ideas.

https://github.com/dstieglitz/daedalum

On Sat, Feb 16, 2019 at 12:04 PM Art Clarke notifications@github.com wrote:

Creating a video player with synced audio and video is actually quite hard (regardless of the programming language or library you use), but is made even harder in Java given that you'll be somewhat the mercy of your garbage collector. Bear that in mind.

But there are a few tricks that are worth bearing in mind:

If your track has both audio and video tracks:

Consider using separate threads for audio and video playback, and give priority to audio feedback. Carefully check your math (and your timebases) to make sure that you can compute how far into the audio has been consumed when you decide to render each video frame. DROP VIDEO FRAMES if you are behind the audio, but keep your prior frame displayed. Check your audio timing before you decode, and again before you render... basically at every point in the video thread if audio has run ahead just move to the next video frame and discard the one you're processing.

If video only, then synchronize to a system time clock rather than the audio frame.

Hope that helps.

In my non open source life, I've been able to do this with HumbleVideo (sorry can't release the code), but I also found it helpful to do the following:

explicitly manage the underlying frame memory (i.e. allocate and free frames). the garbage collector will eventually free humble objects for you, but you won't want to take the pause while it figures out it needs to. I found using an Actor model for demuxing decoding and audio video and audio playback worked well, and I 'synchronized' audio and video based on aligning both the audio actor and the video actor to the system clock, and using pause and resume messages to the video actor if the audio actor was at risk of dropping frames. I used buffering a lot. As in I would try to make sure I had audio decoded and ready to go before I started rendering / starting the video.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[borjaevo]

I did some test buffering decoded audio in a ByteBuffer raw audio array. But method seem to be slowing down video playing is AudioFrame.play(ByteBuffer rawAudio). I will try to run this method in other thread.