MMALResizerComponent output handler supported on video?

[ziriax]

I created a MMALResizerComponent passing a custom implementation of a ICaptureHandler as parameter to its constructor.

However, the Process method is just being called once, or not at all. I found out that ForceStopProcessing was set to `true. I don't want to encoded anything, I want to receive the output the resize component and send it out to UDP (in the long run).

I must be doing something wrong, I'm new to this API.

My code:

    public class CameraUdpStreamer : ICaptureHandler
    {
        public Stopwatch Stopwatch = new Stopwatch();

        public int FrameCount = 0;

        public static readonly ProcessResult NoResult = new ProcessResult();

        public CameraUdpStreamer()
        {
        }

        void IDisposable.Dispose()
        {
        }

        void ICaptureHandler.PostProcess()
        {
            Stopwatch.Stop();
        }

        ProcessResult ICaptureHandler.Process(uint allocSize)
        {
            return NoResult;
        }

        void ICaptureHandler.Process(byte[] data)
        {
            if (FrameCount == 0)
            {
                Stopwatch.Start();
            }
            else
            {
                Debug.Assert(Stopwatch.IsRunning);
            }

            ++FrameCount;
            Console.WriteLine(data.Length);
        }

        string ICaptureHandler.TotalProcessed()
        {
            return $"Processed {FrameCount} frames in {Stopwatch.ElapsedMilliseconds}ms, {FrameCount / Stopwatch.Elapsed.TotalSeconds:000.0}FPS";
        }
    }

    class Program
    {
        static async Task Main(string[] args)
        {
            MMALCameraConfig.Debug = true;
            await TakeVideoManual();
        }

        public static async Task TakeVideoManual()
        {
            MMALCamera cam = MMALCamera.Instance;

            const int fps = 60;

            cam.EnableCamera();

            // https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camera-Module
            MMALCameraConfig.SensorMode = MMALSensorMode.Mode7;
            MMALCameraConfig.VideoResolution = Resolution.As03MPixel;
            MMALCameraConfig.VideoFramerate = new MMAL_RATIONAL_T(fps, 1);

            using (var capturer = new CameraUdpStreamer())
            using (var resizer = new MMALResizerComponent(capturer))
            using (var nullSink = new MMALNullSinkComponent())
            {
                cam.ConfigureCameraSettings();

                var resizerPortConfig = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 640 / 4, 480 / 4, fps, 0, 0, false, null);
                resizer.ConfigureInputPort(MMALEncoding.OPAQUE, MMALEncoding.I420, cam.Camera.VideoPort)
                       .ConfigureOutputPort(resizerPortConfig);

                cam.Camera.VideoPort.ConnectTo(resizer);

                cam.Camera.PreviewPort.ConnectTo(nullSink);

                cam.PrintPipeline();

                // Camera warm up time
                Console.WriteLine("Warming up...");
                await Task.Delay(2000);

                Console.WriteLine("Capturing...");
                var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
                await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);

                Console.WriteLine("Completed!");
            }

            // Only call when you no longer require the camera, i.e. on app shutdown.
            cam.Cleanup();
        }
}

[techyian]

Hi,

I haven't tested hooking up a resizer directly to the video port before but I suspect this behaviour is to be expected as you could simply change the output resolution at the Camera's Video port instead.

Could you try hooking up a splitter component to the video port, then attach the resizer to one of the splitter output ports? Your pipeline would then look like the following:

Camera Video Port -> Splitter -> Resizer -> Optional encoder.

The wiki has a splitter component example here

There is an outstanding bug in the backlog to look at video output when a splitter is in the pipeline so you may run into that hurdle down the line.

What is the reason for wanting to use the resizer component instead of setting your desired resolution at the camera video port level instead?

I will try experimenting this evening to replicate what you're trying to achieve.

[techyian]

I have some news for you, although it's not positive I'm afraid.

The ports linked to components in MMALSharp vary in their behaviour and the choice of port type to use has been hard coded to best match what I felt was correct. This is a problem when a component can be used for both still image capture and also video capture (like the resizer). I am going to have to re-work this, but for the time being if you want to capture data, you'll need to attach a Video Encoder onto the resizer. The splitter component isn't explicitly required for your scenario.

This brings us onto the 2nd issue of the day and it would seem there is a bug with video recording in general, not just when a splitter component is in the pipeline as this ticket suggests. Video playback is much quicker than what is expected. I'll look into that ASAP.

Sorry for the inconvenience. I'll post back when I've got some news for you.

[ziriax]

No hurries, thanks for the info, we are currently using C++ and the raspividyuv.c sample and the flashcam example, these are more low level.

Since we aim for ultra low latency, .net might not have been the best choice, we just wanted to make a quick prototype with your very nice wrapper.

[techyian]

Thanks :)

I've been looking at the video recording bug I mentioned and it looks like it may be an issue on my end as I get the same behaviour with raspivid too (maybe SD card write speed?). So I will focus my efforts on fixing recording directly from the resizer component. If you have a spare moment, would you be kind enough to test video recording with MMALSharp and let me know if you see dropped frame behaviour? Just the standard wiki example will be good enough.

[techyian]

Think I'll have something for you in the next day or two. I've got the functionality working, just need to give it all a good test. Will keep you posted.

[techyian]

Hi,

I've just finished the work for this and merged it into the dev branch. The wiki has been updated with 3 new example pieces of functionality, please see:

Raw capture from resizer

Raw capture from splitter

Raw capture from resizer with splitter

In order to record video from the resizer component, you need to pass in a typeof(VideoPort) to the 2nd parameter of its constructor; this will then listen to the correct buffer output status codes for video capture. If a type isn't passed in, it'll default to still image capture behaviour.

I hope this gives you the functionality you were looking for, let me know if there are any problems. If you need to encode the raw video in the future, you could make use of the example found here which takes video data directly from a stream and encodes it to a video format.

I'll be looking to do an official v0.5.1 release in the next few days as there's a few other bits I want to do before releasing.

Thanks, Ian.

[ziriax]

This is very generous of you!

But we have our project running using a modified raspividyuv.c, so we won't be using .NET/C# on the PI anymore.

But now I know MmalSharp supports this, it might certainly be useful for potential future projects.

[KevinBalz]

Hi,

I tried to implement the example Raw capture from resizer with splitter, but v0.5.1 is not on NuGet yet.

[techyian]

Hey, I'll try and get it pushed onto NuGet this evening. The work is available on the dev branch if you want to grab it from source.

[KevinBalz]

Hey, I considered pulling dev branch too, but with our setup it's a bit cumbersome to integrate it. Thank you for updating nuget in advance!

[techyian]

No problem :) I've updated NuGet with 0.5.1 now - please let me know how you get on with it!

[KevinBalz]

Thank you, works now :) Although I had to replace Fixture.MMALCamera.Camera.VideoPort with cam.Camera.VideoPort, I guess it could be a small documentation error.

[techyian]

Yes you're right! Thanks for spotting that. Glad it's working for you.