search

ffmpeginteropx / FFmpegInteropX

FFmpeg decoding library for Windows 10 UWP and WinUI 3 Apps
Apache License 2.0
211 stars 53 forks source link

Problem with new HW acceleration and stream buffering #230

Closed lukasf closed 2 months ago

lukasf commented 3 years ago

If we want to make our new native HW acceleration the new default, we will have a problem with streaming. The problem is this: Currently, we rely on the MSS buffer to get stable playback in streaming scenarios. The MSS buffer works by buffering our output samples. With the old HW acceleration (passthrough to system decoder), the output are compressed samples in system memory, so that works reasonably well (besides some general bugs in MSS buffering). But with the new HW acceleration, we output decoded frames in GPU memory. Using a buffer on that will quickly blow up GPU memory usage.

If we want to make our approach work with streaming as well, we'd have to provide our own internal buffering implementation.

One approach how this could be done: Instead of reading packets on demand, we could have an async read-ahead packet reader. It would read packets and fill stream buffers until a certain time limit or binary size limit is exceeded. Instead of having streams pull packets on demand, they have to wait until packets are made available.

brabebhin commented 3 years ago

I wonder if we could get the OS to do the buffering for us, by loading ahead in the stream.

lukasf commented 3 years ago

The problem is that FFmpeg does all the stream creation and handling. And none of the buffering options seem to do any effect. There is no general read-ahead mechanism in FFmpeg.

brabebhin commented 3 years ago

This sounds like a consumer-producer problem.

Take a look at this, looks promising...

https://github.com/cameron314/concurrentqueue

lukasf commented 3 years ago

That one could be helpful indeed

brabebhin commented 3 years ago

Unfortunately I won't have much time to look into this in the following weeks...

In the mean time we could do a work around and force system codes when we create a stream source from URL.

lukasf commented 3 years ago

Same for me. I will be very busy the next 5-6 weeks. Been building a house since summer 2020, and it's almost finished now! ^^

brabebhin commented 3 years ago

Sounds cool ^^. Congratulations!

lukasf commented 3 years ago

Thank you. Really looking forward to moving in, maybe in March!! :D But first we need to do the painting and floors ourselves, that will keep us busy for the next weeks...

lukasf commented 2 months ago

Solved by using new stream buffer approach #270