Decode H264 frame to Bitmap

[Mamad4D]

Hi, How can i Decode H264 frame to normal Bitmap?

[BogdanovKirill]

Hello,

You need some code from: https://github.com/BogdanovKirill/RtspClientSharp/tree/master/Examples/SimpleRtspPlayer You could convert DecodedVideoFrames to Bitmap by using something like this:

public Bitmap CopyDataToBitmap(byte[] data)
{
  //Here create the Bitmap to the know height, width and format
  Bitmap bmp = new Bitmap( 352, 288, PixelFormat.Format24bppRgb);  

  //Create a BitmapData and Lock all pixels to be written 
  BitmapData bmpData = bmp.LockBits(
                       new Rectangle(0, 0, bmp.Width, bmp.Height),   
                       ImageLockMode.WriteOnly, bmp.PixelFormat);

  //Copy the data from the byte array into BitmapData.Scan0
  Marshal.Copy(data, 0, bmpData.Scan0, data.Length);

  //Unlock the pixels
  bmp.UnlockBits(bmpData);

  //Return the bitmap 
  return bmp;
}

https://social.msdn.microsoft.com/Forums/vstudio/en-US/e57f7731-c703-4c17-b1a2-32b155f9b745/how-to-convert-byte-array-to-bitmap-in-c?forum=winforms

[BogdanovKirill]

I don't hear any news from you. Feel free to reopen if it is required.

[wangning08115]

@BogdanovKirill This funciton not work fine.

public Bitmap CopyDataToBitmap(byte[] data)

I use this function to convert byte array to Bitmap, but when save bitmap to disk, The image show nothing,only a balck background image.

[wangning08115]

@BogdanovKirill i am sorry , i find this is my mistake.

Below is right code ,work fine. Must set Bitmap width and height to right value.

public System.Drawing.Bitmap CopyDataToBitmap(byte[] data) { //Here create the Bitmap to the know height, width and format System.Drawing.Bitmap bmp = new System.Drawing.Bitmap((int)_writeableBitmap.Width, (int)_writeableBitmap.Height, System.Drawing.Imaging.PixelFormat.Format24bppRgb);

        //Create a BitmapData and Lock all pixels to be written 
        System.Drawing.Imaging.BitmapData bmpData = bmp.LockBits(
                             new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height),
                             System.Drawing.Imaging.ImageLockMode.WriteOnly, bmp.PixelFormat);

        //Copy the data from the byte array into BitmapData.Scan0
        Marshal.Copy(data, 0, bmpData.Scan0, data.Length);

        //Unlock the pixels
        bmp.UnlockBits(bmpData);

        //Return the bitmap 
        return bmp;
    }

[noumanqaiser]

Hi team,

I am planning to use RTSPClientsharp in my windows forms application and want to show image take from a RTSP IP Camera.

I am able to connect to the stream and can see frames arriving in the event handler, however I am not able to figure out how to decode the stream before I can display it in an imageBox. It may sound basic but any help would be appreciated.

Code is attached below.

Regards, Nouman

` public void initializeRTSPSharpClient() { var serverUri = new Uri("rtsp://192.168.1.64:554/ch1/main/av_stream1"); var credentials = new NetworkCredential("admin", "12345");

        var connectionParameters = new ConnectionParameters(serverUri, credentials);
        var cancellationTokenSource = new CancellationTokenSource();

        Task connectTask = ConnectAsync(connectionParameters, cancellationTokenSource.Token);

        //Console.WriteLine("Press any key to cancel");
        //Console.ReadLine();

        //cancellationTokenSource.Cancel();

        //Console.WriteLine("Canceling");
        //connectTask.Wait(CancellationToken.None);

    }

    private  async Task ConnectAsync(ConnectionParameters connectionParameters, CancellationToken token)
    {
        try
        {
            TimeSpan delay = TimeSpan.FromSeconds(5);

            using (var rtspClient = new RtspClient(connectionParameters))
            {
                //rtspClient.FrameReceived +=(sender, frame) => Console.WriteLine($"New frame {frame.Timestamp}: {frame.GetType().Name}");

                rtspClient.FrameReceived += RtspClient_FrameReceived;
                while (true)
                {
                    Console.WriteLine("Connecting...");

                    try
                    {
                        await rtspClient.ConnectAsync(token);
                    }
                    catch (OperationCanceledException)
                    {
                        return;
                    }
                    catch (RtspClientException e)
                    {
                        Console.WriteLine(e.ToString());
                        await Task.Delay(delay, token);
                        continue;
                    }

                    Console.WriteLine("Connected.");

                    try
                    {
                        await rtspClient.ReceiveAsync(token);
                    }
                    catch (OperationCanceledException)
                    {
                        return;
                    }
                    catch (RtspClientException e)
                    {
                        Console.WriteLine(e.ToString());
                        await Task.Delay(delay, token);
                    }
                }
            }
        }
        catch (OperationCanceledException)
        {
        }
    }

    private  void RtspClient_FrameReceived(object sender, RtspClientSharp.RawFrames.RawFrame e)
    {

        //my understanding is some decoding must be done here which I cant figure out how before I 
        //copy the data into bmp, display it and use it to display in an imagebox

        Bitmap bmp = CopyDataToBitmap(e.FrameSegment.Array);
        LatestAcquiredImage = bmp;
        pictureBox.Image = bmp;
        pictureBox.Invalidate();
        imgEntrada = new Emgu.CV.Image<Bgr, byte>(bmp);
    }

    public System.Drawing.Bitmap CopyDataToBitmap(byte[] data)
    {
        //Here create the Bitmap to the know height, width and format
        System.Drawing.Bitmap bmp = new System.Drawing.Bitmap(cameraFrameWidth, cameraFrameHeight, System.Drawing.Imaging.PixelFormat.Format24bppRgb);

        //Create a BitmapData and Lock all pixels to be written 
        System.Drawing.Imaging.BitmapData bmpData = bmp.LockBits(
                             new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height),
                             System.Drawing.Imaging.ImageLockMode.WriteOnly, bmp.PixelFormat);

        //Copy the data from the byte array into BitmapData.Scan0
        Marshal.Copy(data, 0, bmpData.Scan0, data.Length);

        //Unlock the pixels
        bmp.UnlockBits(bmpData);

        //Return the bitmap 
        return bmp;
    }

`

[zsohu]

Hi,

I'm struggling with this same problem. I'm a beginner in C# and cannot get my head wrapped around how I can decode the received raw h264 frames. I put together a really simple test program and I can connect to the camera and start receiving frames. My end goal would be to display those in a PictureBox. I tired to look at your sample in the SimpleRtspPlayer sources but couldn't get what I really need from there. I'm basically at the same place where Nouman were. Could you please help me (us) getting this working?

Best Regards, Zsolt

[carlosjgr7]

quede en lo mismo tengo los frames pero no encuentro como mostrarlos. Ayuda por favor. Saludos Cordiales.

I have the frames but I can't find how to display them. help please. Best Regards,

[Jenscaasen]

Hi, same problem here. The code example seems to only work with "RawJpeg", but at least my camera does not send that. So i am left with RawH264IFrame and RawH264PFrame. Some way to convert H264 to a viewable image format would be handy

[Tafoua]

Hi Hmmm this problem has not been solve .. having same issues. the attach file is a bitmap saved image

[leandrofagundes]

We have already solved it here, it's 6 am now but I tell you when I reach the office later

[Tafoua]

Ok thanks very much .. shall be waiting

On Tue, May 24, 2022, 10:19 AM Leandro Fagundes @.***> wrote:

We have already solved it here, it's 6 am now but I tell you when I reach the office later

— Reply to this email directly, view it on GitHub https://github.com/BogdanovKirill/RtspClientSharp/issues/19#issuecomment-1135633069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIF7B62RTUAA2JOPQIRGMF3VLSNLXANCNFSM4G3JMLMQ . You are receiving this because you commented.Message ID: @.***>

[joaofurlanetto]

This way, it works @Tafoua:

`namespace WPFLiveCam.ViewModels { class MainWindowViewModel { public Command TiraFoto { get; set; } private VideoModel _videoModel; private CaptureModel _captureModel;

    private readonly Action<IDecodedVideoFrame> _invalidateAction;

    public MainWindowViewModel()
    {            
        TiraFoto = new Command(OnTiraFoto);
        _invalidateAction = SaveImagem;

        ReinitializeBitmap();
    }       

    public async void OnTiraFoto(object obj)
    {
        _videoModel = new VideoModel();
        _videoModel.Start(new Uri("rtsp://test............................"));
        var videoSource = _videoModel.VideoSource;
        videoSource.FrameReceived += VideoSource_FrameReceived;
    }

    private const int _width = 1200;
    private const int _height = 800;
    private Int32Rect _dirtyRect;
    private TransformParameters _transformParameters;
    private static readonly Color DefaultFillColor = Colors.Black;
    private static readonly TimeSpan ResizeHandleTimeout = TimeSpan.FromMilliseconds(500);
    private Color _fillColor = DefaultFillColor;
    private WriteableBitmap _writeableBitmap;

    private void ReinitializeBitmap()
    {
        _dirtyRect = new Int32Rect(0, 0, _width, _height);

        _transformParameters = new TransformParameters(
            System.Drawing.RectangleF.Empty,
            new System.Drawing.Size(_width, _height),
            ScalingPolicy.Stretch,
            Project.Velkoz.Decoders.PixelFormat.Bgra32,
            ScalingQuality.FastBilinear);

        _writeableBitmap = new WriteableBitmap(
            _width,
            _height,
            ScreenInfo.DpiX,
            ScreenInfo.DpiY,
            PixelFormats.Pbgra32,
            null);

        RenderOptions.SetBitmapScalingMode(_writeableBitmap, BitmapScalingMode.NearestNeighbor);

        _writeableBitmap.Lock();

        try
        {
            UpdateBackgroundColor(_writeableBitmap.BackBuffer, _writeableBitmap.BackBufferStride);
            _writeableBitmap.AddDirtyRect(_dirtyRect);
        }
        finally
        {
            _writeableBitmap.Unlock();
        }
    }

    private void VideoSource_FrameReceived(object sender, IDecodedVideoFrame decodedVideoFrame)
    {
        if (Application.Current is null)
            return;

        Application.Current.Dispatcher.Invoke(_invalidateAction, DispatcherPriority.Send, decodedVideoFrame);

    }

    private void SaveImagem(IDecodedVideoFrame decodedVideoFrame)
    {
        _videoModel.VideoSource.FrameReceived -= VideoSource_FrameReceived;

        _writeableBitmap.Lock();

        try
        {
            decodedVideoFrame.TransformTo(_writeableBitmap.BackBuffer, _writeableBitmap.BackBufferStride, _transformParameters);

            _writeableBitmap.AddDirtyRect(_dirtyRect);
        }
        finally
        {
            _writeableBitmap.Unlock();
        }

        using FileStream stream5 = new($@"c:\tmp\fotos\{DateTime.Now:dd-MM-yyyy HHmmss}.png", FileMode.Create);
        PngBitmapEncoder encoder5 = new();
        encoder5.Frames.Add(BitmapFrame.Create(_writeableBitmap));
        encoder5.Save(stream5);

        _videoModel.Stop();
    }

    private static unsafe void UpdateBackgroundColor(IntPtr backBuffer, int backBufferStride)
    {
        var DefaultFillColor = Colors.Black;
        var _fillColor = DefaultFillColor;

        byte* pixels = (byte*)backBuffer;
        int color = _fillColor.A << 24 | _fillColor.R << 16 | _fillColor.G << 8 | _fillColor.B;

        Debug.Assert(pixels != null, nameof(pixels) + " != null");

        for (int i = 0; i < _height; i++)
        {
            for (int j = 0; j < _width; j++)
                ((int*)pixels)[j] = color;

            pixels += backBufferStride;
        }
    }        
}

}`

[zoekatakuzinos]

Did anyone solve this issue? I am doing the same thing. I am receiving my frame as a RAWH264PFrame and unable to convert it to display it in a Picture Box on my form (I am in Winforms).

@leandrofagundes my code is like yours.

My picture box display multicolored pixels. Any help would be so appreciated. Tx!

    private  void RtspClient_FrameReceived(object sender, RtspClientSharp.RawFrames.RawFrame e)
    {
        bmp = CopyDataToBitmap(e.FrameSegment.Array);
        lstOutput.Items.Add(String.Format("Received frame {0} timestamp {1}", e.GetType().Name, e.Timestamp.ToString()));
        pictureBox1.Image = bmp;
    }

    public static System.Drawing.Bitmap CopyDataToBitmap(byte[] data)
    {
        //Here create the Bitmap to the know height, width and format
        System.Drawing.Bitmap bmp = new System.Drawing.Bitmap(640, 480, System.Drawing.Imaging.PixelFormat.Format24bppRgb);

        //Create a BitmapData and Lock all pixels to be written 
        System.Drawing.Imaging.BitmapData bmpData = bmp.LockBits(
                             new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height),
                             System.Drawing.Imaging.ImageLockMode.WriteOnly, bmp.PixelFormat);

        //Copy the data from the byte array into BitmapData.Scan0
        System.Runtime.InteropServices.Marshal.Copy(data, 0, bmpData.Scan0, data.Length);

        //Unlock the pixels
        bmp.UnlockBits(bmpData);

        //Return the bitmap 
        return bmp;
    }

[Jenscaasen]

My workaround for this was to start an instance of ffmpeg.exe and get the frame from there, save it to disk and then load the jpg on disk to the picturebox

On Fri, Jan 26, 2024, 11:25 ZoeK @.***> wrote:

Did anyone solve this issue? I am doing the same thing. I am receiving my frame as a RAWH264PFrame and unable to convert it to display it in a Picture Box on my form (I am in Winforms).

@leandrofagundes https://github.com/leandrofagundes my code is like yours.

My picture box display multicolored pixels. Any help would be so appreciated. Tx!

private  void RtspClient_FrameReceived(object sender, RtspClientSharp.RawFrames.RawFrame e)
{
    bmp = CopyDataToBitmap(e.FrameSegment.Array);
    lstOutput.Items.Add(String.Format("Received frame {0} timestamp {1}", e.GetType().Name, e.Timestamp.ToString()));
    pictureBox1.Image = bmp;
}

public static System.Drawing.Bitmap CopyDataToBitmap(byte[] data)
{
    //Here create the Bitmap to the know height, width and format
    System.Drawing.Bitmap bmp = new System.Drawing.Bitmap(640, 480, System.Drawing.Imaging.PixelFormat.Format24bppRgb);

    //Create a BitmapData and Lock all pixels to be written
    System.Drawing.Imaging.BitmapData bmpData = bmp.LockBits(
                         new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height),
                         System.Drawing.Imaging.ImageLockMode.WriteOnly, bmp.PixelFormat);

    //Copy the data from the byte array into BitmapData.Scan0
    System.Runtime.InteropServices.Marshal.Copy(data, 0, bmpData.Scan0, data.Length);

    //Unlock the pixels
    bmp.UnlockBits(bmpData);

    //Return the bitmap
    return bmp;
}

— Reply to this email directly, view it on GitHub https://github.com/BogdanovKirill/RtspClientSharp/issues/19#issuecomment-1911813881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB65EWA3CREJJSV24X4FTR3YQOABLAVCNFSM4G3JMLM2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJRGE4DCMZYHAYQ . You are receiving this because you commented.Message ID: @.***>

[zoekatakuzinos]

Thank you @Jenscaasen for your email. I actually need to stream the video to a picture box or appropriate control first, before taking a snapshot. I have found the SimpleRtspPlayer by @BogdanovKirill (https://github.com/BogdanovKirill/RtspClientSharp) and seeing if I can get that to work. I see it decodes the frames into an image and I am intervening before the image is saved to disk. It is not working yet....I am still struggling a bit, I am getting errors with DLL not found on the FFmpegVideoPInvoke.CreateVideoDecoder call

[zoekatakuzinos]

OK I solved it! Copied the dll's to the debug directory and I am streaming to a picture box. Thank you @BogdanovKirill !!!

[Jenscaasen]

Glad it worked, thank you for posting the solution, so the 3 people per year that stumble on this can be helped :) I don't even remember what I needed this for, 2 years ago, but when i remember i will try your solution as well

On Mon, Jan 29, 2024, 10:05 ZoeK @.***> wrote:

OK I solved it! Copied the dll's to the debug directory and I am streaming to a picture box. Thank you @BogdanovKirill https://github.com/BogdanovKirill !!!

— Reply to this email directly, view it on GitHub https://github.com/BogdanovKirill/RtspClientSharp/issues/19#issuecomment-1914251680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB65EWCWBLQ5CCLCUBO5ZQLYQ5Q3ZAVCNFSM4G3JMLM2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJRGQZDKMJWHAYA . You are receiving this because you were mentioned.Message ID: @.***>