search

ffmpeginteropx / FFmpegInteropX

FFmpeg decoding library for Windows 10 UWP and WinUI 3 Apps
Apache License 2.0
212 stars 53 forks source link

How to play standalone h264 video and pcm audio bytes stream synchronously #286

Closed yikuo123 closed 2 years ago

yikuo123 commented 2 years ago

I receive standalone h264 bytes stream and pcm bytes stream separately, how can I play them together synchronously with MediaElement or MediaPlayerElement? Thanks a lot!

brabebhin commented 2 years ago

Hi

See https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/play-audio-and-video-with-mediaplayer#use-mediatimelinecontroller-to-synchronize-content-across-multiple-players

lukasf commented 2 years ago

I always had the idea to allow adding external audio streams. But until that is actually implemented, the only way is to use MediaTimelineController, as @brabebhin already pointed out.

yikuo123 commented 2 years ago

Thanks! I'll have a try.

I'm looking forward to adding external audio stream, it is what I really want.

lukasf commented 2 years ago

I did not say that I was going to add this anytime soon though. There are some difficulties with that.

Also, if you are receiving these as live streams, then it might be difficult to get them 100% sync, because each stream will have different latency. Both streams would start playback at the current position where they started receiving, with might be slightly different for video and audio stream. I am not sure if it is possible to sync them up correctly in a reliable way, even if we had support for this in our lib. That is why corresponding streams are usually not sent as raw streams, but muxed in a container stream. Or e.g. for DASH format, it has separate streams for audio and video, but each is packed into a MP4 container stream, and that MP4 stream contains the synced up PTS timestamps.

If you work with files, things are much easier. Or if you can make sure that both streams start at the exact same position, then syncing up should be possible.