How to decode MP4 file use GPU?

[wangning08115]

How to decode MP4 file use GPU?

I use the demo of FFmpeg.AutoGen. It is good for code: private static unsafe void DecodeAllFramesToImages() But this is use CPU to decode. I want get a demo use GPU to decode. How can i do this ?

This is FFmpeg.AutoGen demo:

private static unsafe void DecodeAllFramesToImages()
        {
            // decode all frames from url, please not it might local resorce, e.g. string url = "../../sample_mpeg4.mp4";
            var url = "https://rtmp-luzhi.oss-cn-beijing.aliyuncs.com/eryuan2019/%E5%84%BF%E7%AB%A5%E6%89%8B%E8%B6%B3%E5%8F%A3%E7%97%85%E7%9A%84%E9%98%B2%E6%AD%A2%E4%B8%8E%E6%B2%BB%E7%96%97.mp4"; // be advised this file holds 1440 frames
            using (var vsd = new VideoStreamDecoder(url))
            {
                Console.WriteLine($"codec name: {vsd.CodecName}");

                var info = vsd.GetContextInfo();
                info.ToList().ForEach(x => Console.WriteLine($"{x.Key} = {x.Value}"));

                var sourceSize = vsd.FrameSize;
                var sourcePixelFormat = vsd.PixelFormat;
                var destinationSize = sourceSize;
                var destinationPixelFormat = AVPixelFormat.AV_PIX_FMT_BGR24;
                using (var vfc = new VideoFrameConverter(sourceSize, sourcePixelFormat, destinationSize, destinationPixelFormat))
                {
                    var frameNumber = 0;
                    while (vsd.TryDecodeNextFrame(out var frame))
                    {
                        var convertedFrame = vfc.Convert(frame);

                        using (var bitmap = new Bitmap(convertedFrame.width, convertedFrame.height, convertedFrame.linesize[0], PixelFormat.Format24bppRgb, (IntPtr) convertedFrame.data[0]))
                            bitmap.Save($"frame.{frameNumber:D8}.jpg", ImageFormat.Jpeg);

                        Console.WriteLine($"frame: {frameNumber}");
                        frameNumber++;
                    }
                }
            }
        }

[mariodivece]

You can check out how it's done in this project -- it's slightly complex but you can follow through the sample app and step through the code. You will notice that you need to create a GPU device context, send the frame containing packet data to the GPU and then transfer the frame from the GPU and on to the RAM -- after that, you will need to convert the frame from GPU specific colorspace to a colorspace that will work for you.

Search my code for the following function. The key is to examine the ExchangeFrame call to the HardwareAccelerator object.

protected override MediaFrame CreateFrameSource(IntPtr framePointer)

Now, what I described is one way of doing it -- there is also a simpler way of doing it (if the codec is available directly) -- Example:

// C/C++ example to use the DXVA2 decoder
AVCodec* decoder = avcodec_find_decoder_by_name ("h264_dxva2");
avcodec_open2 (decoder_ctx, decoder, NULL);

As far as providing sample code for your specific purposes, I apologize but I would consider this outside the scope of this project.