How To Get Live Frame By Frame Context For Video Output

[uastephen45]

First this library is awesome!

I'm trying to stream frame by frame of a video into memory to send to a client. I started by using the Camera.TakePicture method and would stream the output into a memory stream.However this is very slow for trying to get 30fps. And i don't want to do Camera.TakeVideo because i have to wait until the recording is done to send the data to the client. is there a way to do Camera.TakeVideo and get context to each frame as they are captured live?

Thanks for the support!

[techyian]

Hi, thanks for the feedback :)

Yes, I think I'm going to have to make this a bit clearer in the docs as it's a common question at the moment.

There are two things which will help you achieve what you're looking for. First, you want to use the rapid capture functionality which will allow you to attach an image encoder to the camera's video port, an example of this can be found here. Next, you want to create your own capture handler, subclassing InMemoryCaptureHandler and that will allow you to intercept calls to the Process method.

Could you let me know if the below suits your needs? You can change the working framerate of the camera's video port using the config MMALCameraConfig.VideoFramerate. This is constrained by the resolution you set the video port too also, see here or here depending on your camera version.

public class MyInMemoryCaptureHandler : InMemoryCaptureHandler
{        
    public override void Process(byte[] data, bool eos)
    {
        // The InMemoryCaptureHandler parent class has a property called "WorkingData". 
        // It is your responsibility to look after the clearing of this property.

        // The "eos" parameter indicates whether the MMAL buffer has an EOS parameter, if so, the data that's currently
        // stored in the "WorkingData" property plus the data found in the "data" parameter indicates you have a full image frame.

        // The call to base.Process will add the data to the WorkingData list.
        base.Process(data, eos);

        if (eos)
        {
            Console.WriteLine("I have a full frame. Clearing working data.");
            this.WorkingData.Clear();                
        }
    }
}

public async Task TakePictureFromVideoPort()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var myCaptureHandler = new MyInMemoryCaptureHandler())
    using (var splitter = new MMALSplitterComponent())
    using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();

        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, myCaptureHandler);

        cam.Camera.VideoPort.ConnectTo(splitter);
        splitter.Outputs[0].ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);

        // Camera warm up time
        await Task.Delay(2000);

        CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));

        // Process images for 15 seconds.        
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

[techyian]

What encoding and pixel format is the client expecting the frames to be in?

The example above produces encoded image frames as JPEGs using YUV420, but you could also do something similar with video too if you wanted. My reply to this question might help you? The OP wanted raw frames using a BGR24 pixel format, but you could tweak the solution to match what your client is expecting.

[uastephen45]

You Sir Rock! this is perfect!

here is my code for anyone else reading.

public class ActiveVideoInMemoryHandler : InMemoryCaptureHandler, IVideoCaptureHandler { CPacheStream cPacheStream; public ActiveVideoInMemoryHandler(CPacheStream cpacheStream) { this.cPacheStream = cpacheStream; } public override void Process(byte[] data, bool eos) { Console.WriteLine("I'm in here"); base.Process(data, eos); if (eos) { //Broadcast to interested network streams cPacheStream.Broadcast(Convert.ToBase64String(this.WorkingData.ToArray())); this.WorkingData.Clear(); Console.WriteLine("I have a full frame. Clearing working data."); } }

    public void Split()
    {
        throw new NotImplementedException();
    }

}

 public static async Task TakeRawVideo(CPacheStream cPacheStream)
    {
        // By default, video resolution is set to 1920x1080 which will probably be too large for your project. Set as appropriate using MMALCameraConfig.VideoResolution
        // The default framerate is set to 30fps. You can see what "modes" the different cameras support by looking:
        // https://github.com/techyian/MMALSharp/wiki/OmniVision-OV5647-Camera-Module
        // https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camera-Module            
        using (var myCaptureHandler = new ActiveVideoInMemoryHandler(cPacheStream))
        using (var splitter = new MMALSplitterComponent())
        using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))
        using (var nullSink = new MMALNullSinkComponent())
        {
            cam.ConfigureCameraSettings();
           var portConfig = new MMALPortConfig(MMALEncoding.JPEG,MMALEncoding.I420, 90);
            // Create our component pipeline.         
            imgEncoder.ConfigureOutputPort(portConfig, myCaptureHandler);
            cam.Camera.VideoPort.ConnectTo(splitter);
            splitter.Outputs[0].ConnectTo(imgEncoder);
            cam.Camera.PreviewPort.ConnectTo(nullSink);
            // Camera warm up time
            await Task.Delay(2000);
            CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));
            // Process images for 15 seconds.        
            await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
        }
    }