Motion detection plug-in model and analysis

[MV10]

And here we go again!

Improved summed RGB diff logic

• Use absolute value in the comparison (bugfix)
• Separate RGB diff threshold from frame-wide change-count threshold
• Switch to cell-based frame-level change counts
• Per-cell threshold as a percentage of pixels in the cell

New Stuff

• Separate motion detection from motion detection frame buffer handling
• A plug-in approach to the motion detection algorithm
• Optional real time video analysis output from motion algorithms
• Ability to leave motion detection running while suppressing the detection callback
• Improved test frame update logic ("cool down" avoids updating during motion)

Capture Handlers

• IMotionCaptureHandler.DisableMotionCapture adds a new argument onlyDisableCallback
• FrameBufferCaptureHandler implements new argument and has renamed FrameDiffDriver
• MotionAnalysisCaptureHandler writes full raw frames to a file stream, motion algorithm callback-based

Frame Buffer Handling

• FrameDiffDriver handles frame buffers and invoking the selected motion detection algorithm
• FrameDiffMetrics is a thread safe struct used to pass parameters to the motion algorithm
• IMotionAlgorithm defines how the frame driver interacts with the algorithms
• MotionAlgorithmBase provides a couple of utility methods
• MotionAlgorithmRGBDiff is the parallel processing motion algorithm and analysis code
• FrameDiffAnalyser is gone (replaced by FrameDiffDriver and IMotionAlgorithm)

Configuration

• The selected algorithm is referenced by the MotionConfig.MotionAlgorithm property
• Pass algorithm-specific parameters to the algorithm constructor
• Threshold is removed from MotionConfig

Example

using var motionCaptureHandler = new FrameBufferCaptureHandler();
using var resizer = new MMALIspComponent();

resizer.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
resizer.ConfigureOutputPort<VideoPort>(0, new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480), motionCaptureHandler);
cam.Camera.VideoPort.ConnectTo(resizer);

await Task.Delay(2000); // camera warmup

var cts = new CancellationTokenSource(TimeSpan.FromSeconds(totalSeconds));

// These are the defaults
var motionAlgorithm = new MotionAlgorithmRGBDiff(
        rgbThreshold: 200,
        cellPixelPercentage: 50,
        cellCountThreshold: 20
    );

// Optional, outputs modified frames (other algorithms could do other things?)
// For RGBDiff algorithm, output goes nowhere without the callback
// MotionAnalysisCaptureHandler does this internally, writes output to stream
motionAlgorithm.EnableAnalysis(analysisFramebufferCallback: null);

// These are the defaults
var motionConfig = new MotionConfig(
        algorithm: motionAlgorithm, 
        testFrameInterval: TimeSpan.FromSeconds(3), 
        testFrameCooldown: TimeSpan.FromSeconds(3)
    );

await cam.WithMotionDetection(
    motionCaptureHandler,
    motionConfig,
    async () =>
    {
        // callback code omitted... 
    }
    .ProcessAsync(cam.Camera.VideoPort, cts.Token);

[MV10]

A small but unrelated change that I'd like to commit, if you don't mind, is to activate the "generate package on build" option on the csproj files. This makes it much easier to work with the library in my separate test program.

I'm also interested in how you address that problem yourself. In my case, I have a dev_packages folder as a NuGet package source, and I've simply written a batch file that copies the nupkg files there and removes the packages from the user-directory .nuget cache. It's a bit inelegant though, because I have to remember to run the batch and also to close VS and re-open to get it to re-load the updated packages -- unloading and reloading the solution doesn't seem to be enough.

[MV10]

Just thinking out loud (I know you're busy right now):

I am going to modify IMotionAlgorithm to use an output handler as the callback. The only oddity is that ImageContext is typically buffer-based throughout the library, whereas this will be a full raw frame (though low-res buffers are often full frame, of course). I need to think about how to populate some of the context properties (probably via FrameDiffMetrics).

void EnableAnalysis(IOutputCaptureHandler handler = null);

While I really wanted to run the analysis output (and other filtering) earlier in the pipeline so that the hardware could do h.264 against the filtered output, that isn't looking realistic. But this change still allows for streaming by piping the raw frames through ExternalProcessCaptureHandler and the helper in VLCCaptureHandler.

The new (in this PR) class MotionAnalysisCaptureHandler becomes unnecessary.

I'm also wondering if dishing out full frames to ffmpeg will have any effect on its ability to output a valid MP4. Probably not, but I'll give it a shot.

[MV10]

Figures... according to the muxer matrix, VLC won't convert raw frames to MJPEG.

https://www.videolan.org/streaming-features.html

So ... into the ffmpeg switch-swamp I go. Wish me luck.

Annnnd... ffmpeg doesn't work as a server like VLC does. 🙄

Getting pretty close, I had the idea of using ffmpeg to encode the RGB24 to h.264, then pipe that into VLC for streaming. After many hours of work I have it close to working. I put the commands into a script with this beast of a command line:

#!/bin/bash
ffmpeg -hide_banner -f rawvideo -c:v rawvideo -pix_fmt rgb24 -s:v 640x480 -r 24 -i - -f h264 -c:v h264_omx -b:v 2500k - | cvlc stream:///dev/stdin --sout "#transcode{vcodec=mjpg,vb=2500,fps=24,acodec=none}:standard{access=http{mime=multipart/x-mixed-replace;boundary=7b3cc56e5f51db803f790dad720ed50a},mux=mpjpeg,dst=:8554/}" :demux=h264

From a terminal I can do something like this to create a stream which I can browse to:

cat /media/ramdisk/input.raw | ./ffmpeg2clvc.sh

When I plug the script into ExternalProcessCaptureHandler I can see that it launches but it seems that it doesn't receive data. I suspect this is because my ffmpeg command is using the h264_omx hardware encoder but my own MMALSharp program has that tied up already...

[MV10]

I overlooked your reply to #108 last week (image processing performance), but I'm starting to think FrameAnalyser would be a more sensible place for some of these changes (collecting frame metrics, storing the list of rectangles for parallel cell-based processing, etc.) This way the motion-diff class would only be managing the test frame, the new comparison frame, and the diff algorithm. Whereas FrameAnalyser would be reusable by the static image processing, or perhaps even other places in the pipeline.

I've also read that Gaussian blur is helpful for motion detection noise-removal, so once I feel this PR is ready to roll, I will probably tackle that next.

[MV10]

Got it working ... this is a frame from my browser of realtime 24FPS filtered motion analysis (it's pointing elsewhere in my office and picking up the news on TV). As described earlier, /bin/bash is the external process, the raw RGB24 stream pipes into ffmpeg which turns it into an h.420 YUV420p stream (that format is what I was missing the other day) and pipes that to CLVC which demuxes into an MJPEG HTTP stream and listens on the port. Pretty cool!

static async Task visualizeStream(int seconds)
{
    var cam = GetConfiguredCamera();

    MMALCameraConfig.Resolution = new Resolution(640, 480);
    MMALCameraConfig.SensorMode = MMALSensorMode.Mode7; // for some reason mode 6 has a pinkish tinge
    MMALCameraConfig.Framerate = new MMAL_RATIONAL_T(20, 1);

    Console.WriteLine("Preparing pipeline...");
    cam.ConfigureCameraSettings();

    var motionAlgorithm = new MotionAlgorithmRGBDiff(
            rgbThreshold: 200,          // default = 200
            cellPixelPercentage: 50,    // default = 50
            cellCountThreshold: 20      // default = 20
        );

    var motionConfig = new MotionConfig(
            algorithm: motionAlgorithm,
            testFrameInterval: TimeSpan.FromSeconds(3), // default = 3
            testFrameCooldown: TimeSpan.FromSeconds(3)  // default = 3
        );

    var raw_to_mjpeg_stream = new ExternalProcessCaptureHandlerOptions
    {
        Filename = "/bin/bash",
        EchoOutput = true,
        Arguments = "-c \"ffmpeg -hide_banner -f rawvideo -c:v rawvideo -pix_fmt rgb24 -s:v 640x480 -r 24 -i - -f h264 -c:v libx264 -preset ultrafast -tune zerolatency -vf format=yuv420p - | cvlc stream:///dev/stdin --sout '#transcode{vcodec=mjpg,vb=2500,fps=20,acodec=none}:standard{access=http{mime=multipart/x-mixed-replace;boundary=7b3cc56e5f51db803f790dad720ed50a},mux=mpjpeg,dst=:8554/}' :demux=h264\"",
        DrainOutputDelayMs = 500, // default = 500
        TerminationSignals = ExternalProcessCaptureHandlerOptions.SignalsFFmpeg
    };

    using (var shell = new ExternalProcessCaptureHandler(raw_to_mjpeg_stream))
    using (var motion = new FrameBufferCaptureHandler(motionConfig, null))
    using (var resizer = new MMALIspComponent())
    {
        // Argument is a capture handler, this is the bash/ffmpeg/clvc pipeline
        motionAlgorithm.EnableAnalysis(shell);

        // Not taking action when motion is detected in this example,
        // but the resizer feeds normal motion detection
        resizer.ConfigureOutputPort<VideoPort>(0, new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480), motion);
        cam.Camera.VideoPort.ConnectTo(resizer);

        await cameraWarmupDelay(cam);

        Console.WriteLine($"Streaming MJPEG with motion detection analysis for {seconds} sec to:");
        Console.WriteLine($"http://{Environment.MachineName}.local:8554/");
        var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(seconds));
        await Task.WhenAll(new Task[]{
                shell.ProcessExternalAsync(timeout.Token),
                cam.ProcessAsync(cam.Camera.VideoPort, timeout.Token),
            }).ConfigureAwait(false);
    }

    cam.Cleanup();

    Console.WriteLine("Exiting.");
}

[MV10]

AppVeyor can't create the ffmpeg nuget package? Looks like it skipped building the ffmpeg project, for some reason.

[techyian]

This looks to be because you've added GeneratePackageOnBuild to true, please see here. To be honest, I'd rather this be removed. Locally I just reference the DLLs in my test project if there's stuff that hasn't been pushed to GitHub. The AppVeyor setup has worked well for a few years now and you can always add the MyGet repository for work that's been committed to the dev branch.

[MV10]

Oops, meant to pull those out before check-in ... fixing it.

[MV10]

I don't think I listed the specific changes in that last commit:

• MotionAnalysisCaptureHandler removed, no longer required
• MotionAlgorithmRGBDiff altered to output analysis-mode images to a capture handler
• Added StreamRawRGB24asMJPEG to the VLC helper
• Renamed the existing MJPEG streaming helper to StreamH264asMJPEG

If you agree with my work described in #108 and we figure out the raw capture / manipulate situation, these are the changes versus this original PR's description that I will commit. I wanted to get them written down while it's still reasonably fresh in my mind, I suspect I'm about to get swamped by my day job, too.

Frame and Motion Processing

• FrameAnalyser no longer abstract, handles first-frame metadata collection, owns parallel processing cells
• FrameDiffMetrics struct renamed to FrameAnalysisMetadata (thought it was more clear that way)
• The cell divisor was separated into horizontal and vertical counts, as explained in #108
• FrameDiffDriver scaled back to just managing the mask, test frame, and invoking analysis

Matrix Convolution

• ConvolutionBase requires raw frames, uses FrameAnalyser, uses cell-based parallel processing
• ConvolutionBase uses an ImageContext as input
• If the context StoreFormat property is null (via Manipulate), the convolution output is a raw frame
• All related classes have a constructor override to accept custom cell-count values
• BoxBlur processor added
• LineDetection processor added (directional edge detection)
• Minor EdgeDetection changes to make it more like the other processing classes (fewer public members)

MMALSharp.Common

• ImageContext properties changed to fields for hot-path performance reasons
• FormatRawImageExtension adds ImageContext.FormatRawImage for raw Data to JPEG, BMP, etc

None of the above changes usage.

Again, just wanted to document everything that changed, no rush at all, having this PR open isn't blocking me or anything. I know the other thread is long and this is a lot to review!

Have a great weekend, we're taking the dog to the bar and doing absolutely nothing useful at all! 😁

[MV10]

Ian, thanks for taking time to look. I was sort of waiting for your feedback about changing ImageContext properties to fields. The benefits are significant, but that's a deviation from your general policy for public scope members. If you're OK with it based on the test results, I'll commit the final round of changes after looking at that constructor you mentioned above.

And yessir, I definitely agree, I have quite a lot of documentation work lined up!

[MV10]

Oh and I'm also still unsure of how to set up the camera to get one raw frame into the handler -- it exits before the image processor is able to finish running. See this comment:

https://github.com/techyian/MMALSharp/issues/108#issuecomment-689754922

[MV10]

Figured it was silly not to just merge the convolution changes, too. Everything is easy enough to roll back if necessary.

(The merge conflict was just around some stuff you'd changed in ConvolutionBase -- which is almost completely changed in my commit.)

[MV10]

EDIT -- In case you read the original by email -- I realized I was thinking about that wrong (distracted by yesterday's BGR/RGB tests that used Bitmap.Save to a file, whereas convolution saves to a memory stream) -- but this really should be a stand-alone issue, so I'll rewrite it.

[techyian]

Hi Jon, I've merged this PR now so thank you for your efforts on this. It would be great when you get some spare time if you could produce some documentation to surround this work and some useful examples. Now that this is merged I will begin looking at the tickets created off the back of this PR.

Thanks, Ian

[MV10]

Fantastic! Yes, I will get to work on the wiki soon. Thanks for all the help and patience with this one.

[MV10]

Quick-pass wiki updates -- convolutions (the low-hanging fruit for documenting this PR):

https://github.com/techyian/MMALSharp/wiki/Image-Processing

...and a note that motion detection changes a lot in v0.7:

https://github.com/techyian/MMALSharp/wiki/Advanced-Examples#FrameDiffDetection

Finally, I started that new CCTV page, which I will start with motion detection, then get into still frame captures, the various options, real-time analysis output, etc. -- just a placeholder today, but I'll try to work on this a bit each afternoon:

https://github.com/techyian/MMALSharp/wiki/CCTV-and-Motion-Detection

Do you know if it's possible to store images in the wiki itself? I thought of a trick that might work -- open a new issue, paste the image there (which uploads it to github) then reference that URL from the wiki. The issue can be closed but new images can still be added later...

[MV10]

Quite a bit of content added today to the motion/CCTV wiki... (had a work day that involved mostly waiting on other people!)

[techyian]

Very impressed with the write up, Jon, so thank you for that. Although I've been following the work you've been doing lately, seeing it all written down has really helped me understand the full picture of what you've been working on. I've re-jigged the tickets in the v0.7 milestone so I think once the motion stuff is completed and other minor bits, I'll look at pushing a new official release out. Great job.

[techyian]

Do you know if it's possible to store images in the wiki itself?

I think it would probably be easier to create a resources (or similarly named) folder containing all the images used in the wiki. Send a PR in if you want and then you can reference them in your docs.

[MV10]

Thanks. I know I can be hard to follow when I'm in the thick of it, but I'm still really happy about finding your library and your willingness to consider some pretty major changes.

I went ahead and finished it up and I did reference my one motion analysis image from this PR, which I will probably point to a youtube capture of motion analysis at some point. Longer term I think there might be some benefit in diagrams elsewhere in the wiki to help explain component relationships and that sort of thing (I'm trying to think about stuff I had some trouble understanding initially), so I'll open a ticket when I'm feeling like I'm in a documenting mood. (It comes and goes -- and after writing that monster CCTV section, it's very gone right now!)

I have some other ideas to further improve motion detection but I think the Really Big Stuff is settled.

Have a good one, talk to you soon.