New FrameBufferCaptureHandler and revised CircularBufferCaptureHandler

[MV10]

Fixes #163

As discussed, FrameBufferCaptureHandler either writes on-demand images with a call to WriteFrame when wired up to a continuous-capture image encoder, or handles motion detection processing when wired up to a video encoder. As a result, CircularBufferCaptureHandler is pared back to simply buffering video.

There's no difference in the usage of CircularBufferCaptureHandler as far as video recording goes, but I'll follow up with another post containing an example of doing motion capture (it's exactly the same) and on-demand images.

Glad to see this one wrapping up!

[MV10]

Here's an example of usage from my own test program -- splitter-based so it performs motion detection and image capture, as well as h.264 recording. Once this is merged I'll update the wiki with examples that are more similar to the existing examples. This one is set up to call out the changes for your review.

var cam = GetConfiguredCamera();
MMALCameraConfig.InlineHeaders = true;
cam.ConfigureCameraSettings();

// empty constructor for motion detection
using (var motionCaptureHandler = new FrameBufferCaptureHandler())

// path and extension constructor for image capture
using (var imgCaptureHandler = new FrameBufferCaptureHandler(ramdiskPath, "jpg"))

// this will feed the image capture handler above
using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))

// typical for a splitter example
using (var vidCaptureHandler = new CircularBufferCaptureHandler(4000000, ramdiskPath, "h264"))
using (var splitter = new MMALSplitterComponent())
using (var resizer = new MMALIspComponent())
using (var vidEncoder = new MMALVideoEncoder())
using (var renderer = new MMALVideoRenderer())
{
    // typical for a splitter example
    splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
    resizer.ConfigureOutputPort<VideoPort>(0, new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480), motionCaptureHandler);
    vidEncoder.ConfigureOutputPort(new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 0, MMALVideoEncoder.MaxBitrateLevel4, null), vidCaptureHandler);

    // send image encoder output to the image capture handler
    imgEncoder.ConfigureOutputPort(new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, quality: 90), imgCaptureHandler);

    // typical for a splitter example
    cam.Camera.VideoPort.ConnectTo(splitter);
    cam.Camera.PreviewPort.ConnectTo(renderer);
    splitter.Outputs[0].ConnectTo(resizer);
    splitter.Outputs[1].ConnectTo(vidEncoder);
    splitter.Outputs[2].ConnectTo(imgEncoder);

    // camera warmup
    await Task.Delay(2000);

    // processing for 30 seconds
    var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));

    // typical for a motion capture sample
    var motionConfig = new MotionConfig(threshold: 130, testFrameInterval: TimeSpan.FromSeconds(3));

    // prepare the camera and begin processing
    await cam.WithMotionDetection(
        motionCaptureHandler,
        motionConfig,
        async () =>
        {
            // this lambda is invoked when motion is detected

            // stop detecting motion
            motionCaptureHandler.DisableMotionDetection();

            // save a snapshot as soon as motion is detected
            imgCaptureHandler.WriteFrame();

            // register a lambda to execute when we're done recording video
            var stopRecordingCts = new CancellationTokenSource();
            stopRecordingCts.Token.Register(() =>
            {
                // resume detecting motion, stop recording, and start a new video file
                motionCaptureHandler.EnableMotionDetection();
                vidCaptureHandler.StopRecording();
                vidCaptureHandler.Split();
            });

            // save additional snapshots 1- and 2-seconds after motion is detected
            var stillFrameOneSecondCts = new CancellationTokenSource();
            var stillFrameTwoSecondsCts = new CancellationTokenSource();
            stillFrameOneSecondCts.Token.Register(imgCaptureHandler.WriteFrame);
            stillFrameTwoSecondsCts.Token.Register(imgCaptureHandler.WriteFrame);

            // set token cancellation timeouts
            stopRecordingCts.CancelAfter(5 * 1000);
            stillFrameTwoSecondsCts.CancelAfter(2000);
            stillFrameOneSecondCts.CancelAfter(1000);

            // record video
            await Task.WhenAny(
                    vidCaptureHandler.StartRecording(vidEncoder.RequestIFrame, stopRecordingCts.Token),
                    cts.Token.AsTask()
            );

            // ensure all tokens are cancelled if the overall cts.Token timed out
            if (!stopRecordingCts.IsCancellationRequested)
            {
                stillFrameOneSecondCts.Cancel();
                stillFrameTwoSecondsCts.Cancel();
                stopRecordingCts.Cancel();
            }
        })
        .ProcessAsync(cam.Camera.VideoPort, cts.Token);
}

cam.Cleanup();

[techyian]

One thing I have noticed which happens randomly whilst running motion detection are errors such as this being thrown:

Unhandled exception. System.ObjectDisposedException: Cannot access a closed file.
   at System.IO.FileStream.ValidateReadWriteArgs(Byte[] array, Int32 offset, Int32 count)
   at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
   at MMALSharp.Handlers.CircularBufferCaptureHandler.Process(ImageContext context) in C:\Users\Ian\source\repos\MV10\src\MMALSharp.Processing\Handlers\CircularBufferCaptureHandler.cs:line 128
   at MMALSharp.Callbacks.PortCallbackHandler`2.Callback(IBuffer buffer) in C:\Users\Ian\source\repos\MV10\src\MMALSharp\Callbacks\PortCallbackHandler.cs:line 81
   at MMALSharp.Callbacks.VideoOutputCallbackHandler.Callback(IBuffer buffer) in C:\Users\Ian\source\repos\MV10\src\MMALSharp\Callbacks\VideoOutputCallbackHandler.cs:line 121
   at MMALSharp.Ports.Outputs.VideoPort.NativeOutputPortCallback(MMAL_PORT_T* port, MMAL_BUFFER_HEADER_T* buffer) in C:\Users\Ian\source\repos\MV10\src\MMALSharp\Ports\Outputs\VideoPort.cs:line 90
Aborted

In the CircularBufferCaptureHandler there's no protection around the calls to this.CurrentStream.Write, I think they need a if (this.CurrentStream.CanWrite) block surrounding it as seen in the StreamCaptureHandler class. If you're happy to add to this PR I'd appreciate it! Is it also worth adding null checks too?

[MV10]

I'll go through it and add some protections, as well as the changes discussed in #171.

I haven't seen any exceptions. I'm trying to imagine how or why that might occur. We're about to leave the house, I'll put my test program into a motion detection loop for several hours and see whether it happens for me. My recent testing tends to have bene short (30, 60, 90 seconds).

[MV10]

So 22 motion events over a 3 hour period each recorded a 10 second clip with no exceptions. I suppose I should try some high-frequency clip splitting (like timer-based instead of motion-driven)? Maybe it's a race condition somewhere? Is your test short enough to post?

[MV10]

I see that StreamCaptureHandler throws if the stream is not writeable. I suppose the question is whether we should just fail silently in the circular buffer?

I'm writing to ramdisk, maybe writing to SD is slow enough that it sometimes writes frame data while it's still in the process of creating a new file after Split? I guess a callback is firing to make that happen? I don't see that anything ever sets CurrentStream to null, but I suppose the callback might fire before the constructor finishes?

[techyian]

I'm using the motion detection example from the wiki. I have a suspicion that the background thread spun up by the native callback method is still active and hitting the capture handler's Process method at the point in which the cancellation token passed to ProcessAsync expires and the capture handlers begin to be tore down. That would explain the randomness as the timing would need to be very precise. Adding in the extra protection around CurrentStream.Write() will hopefully be enough to sort it.

[MV10]

That makes sense. I wonder though, will calling CanWrite trigger the System.ObjectDisposedException?

Also, I've just noticed both else blocks in Process do the same thing, apart from setting _receivedIFrame -- probably a left-over side-effect of the bits that were removed. I'll add checks and clean that up.

[MV10]

Ok, all up to date with dev again.

I'm starting to think we ought to wait on the additional motion-detection-related changes discussed in the other PR (making threshold early-exit optional in MotionConfig, the diff interface, etc.) -- I want to see whether I can actually figure out useful cell-proximity features before we go complicating the existing code any further.

Also, as it stands now, I have a fair bit of work ahead of me to get the Wiki updated for these most recent 3 PRs.

What do you think?

[techyian]

Thanks for providing the example app to test against, that really helped. It appears to be working as expected and the extra safety checks also seem to have helped with the circular buffer, too :) With regards to the wiki, I think it would be beneficial to introduce your example application as "advanced" functionality, perhaps even in its own sub-section to demonstrate how to take image stills once motion has been detected, what do you think? We could even have a dedicated CCTV sub-section?

Anyway, let's get this merged!

[MV10]

Thanks for merging this.

Yes, I agree this qualifies as advanced usage (thinking back to my first encounter with MMALSharp).

I like the idea of a CCTV section. The splitter is an important feature for that scenario -- still captures, motion detection, recording, streaming for live monitoring... can't think of a good use for the fifth output but I'm sure something will come up.

Now that I'm more familiar with the library, I think what would be helpful in the wiki is to catalog the different components and how they relate to the pipeline. The "Handling Data" topic initially made me feel as if I understood the pipeline, but when it comes to the coding I find I'm still a bit confused about how they all talk to each other. I think the topic is clear and well written, but it needs more detail. Given the number of components in the library, that may warrant a separate section. I know it's a lot of work, I'd offer to help but I don't understand the pipeline well enough yet, so call this a suggestion.

I'll get all this pulled to my own machine and get the wiki changes into place in the next few days. Thanks again.

[MV10]

I find I'm still a bit confused about how they all talk to each other

As an example, there are places where you pass null to capture handler arguments (I think it was in a splitter output example), and recently you replied to one of my comments with sample code where components wired themselves together automatically -- how and when that works is unclear.

Trying to provide feedback here as an "outsider" -- I know what it's like to build a big library where you know all the parts, and it can be hard to figure out what is and isn't obvious to newcomers. (I'm actually doing that at work right now, been working on the foundational bits of a giant new system for close to a year, rewriting a financial platform I built 20 years ago -- and within the next few months we'll finally be pulling other devs into the project, so I guess this factor has been on my mind a lot.)

[techyian]

Thanks, I'll take that feedback onboard and see what I can do. Could you raise a new issue please and document your concerns then that will give me a basis to work off? There are still times when I wonder why something works in a particular way, especially when working with the native MMAL bits and bobs, and sometimes different components react to a pipeline configuration in a different way to other components (as you saw with the splitter). I'm by no means an MMAL expert and have on many occasions had to go to the Raspberry Pi forums for official support. I have also been reluctant to go into great depth with documenting the MMAL side of things due to the extensive write-up done by Dave Jones for the picamera project, but I can certainly try and piece together what I know.