Decode Live H264 Byte Arrays

[briotto-deloitte]

Hi, I am trying to decode raw h264 data that is being sent as part of FlatBuffer messages sent over WebSocket (this is required because there is other information about those frames embedded within the overall message). Because of that, I essentially need to create a method that takes in chunks (byte[]) of h264 data and feeds them to a decoder that will process it into a displayable format. Overall, this feels similar to https://github.com/microsoft/FFmpegInterop/issues/165 and I've tried to implement a solution based off of that. My current process is to

• Extract the h264 byte[]
• Write it to a MemoryStream
• Get an IRandomAccessStream from that using AsRandomAccessStream()
• If it's the first frame received, it will create a FFmpegInteropMSS using CreateFromStreamAsync

However, I am running into the same issue that that person was running into where it will play whatever is in that first packet and then stalling. I assume this is because of what was mentioned here

You absolutely cannot use InMemoryRandomAccessStream for your scenario. As I wrote, you need to get the IInputStream from your WebSocket. Then have ffmpeg read directly from that IInputStream, by providing a custom FileStreamRead implementation. Only this will make sure that ffmpeg will not read beyond what is available.

I also see this as a suggestion

After thinking about it some more, the easiest thing that might be done would be to mimic the way we wrap the RandomAccessStream into an IStream interface and wrap the WebSocket inputstream. We could then have a CreateMediaStreamSource taking a WebSocket as an input parameter.

I could also resend the data to myself over TCP and call CreateFFmpegInteropMSSFromUri on that, but that seems less than ideal for many reasons. Based on my situation, what would be the best path forward?