Non-Standard Image Size Being Cut Off

[d8ahazard]

Hello again!

So, I was finally able to start testing video capture and processing on the raspberry Pi, and I'm now encountering a new issue.

For a camera module, I am using the Innomaker OV5647 sensor. It has a maximum listed resolution of 2592 x 1944.

However, when I try to capture video at any resolution, the sides of the image are cut off. From capture:

From raspistill:

I don't need the video to be at full resolution - in my tests I'm setting it to half the value. But is there some way to tell it to use the full width of the camera's capture abilities so it's not cut off on the sides?

[d8ahazard]

Update:

Just wanted to make sure that raspivid was able to grab "full" width. Using the same scale I'm trying to test with in my app, I was able to grab this using raspivid:

[techyian]

Hi!

By default, videos captured in MMALSharp are taken at 1920x1080 (see here), and if you check the documentation you will see that this will use a partial field of view which coincides with what you're seeing.

It's important to note that the resolutions mentioned in the documentation are the resolutions the sensor can directly output to the Pi's GPU, however you can request a lower resolution and the GPU will resize on your behalf. You will see in the docs that each resolution the camera module directly outputs is assigned a "mode"; this mode is automatically selected by default (Mode 0) based on the resolution and framerate selected. You can also force the sensor to use a specific sensor mode in case it chooses the wrong one automatically by setting the config MMALCameraConfig.SensorMode, see the docs here.

Any configuration changes should be set before ConfigureCameraSettings() is called in your application.

Hopefully that explains what's going on and will help get you sorted.

[d8ahazard]

I follow what you're saying, but the output is still cut off? I can read the cam mode after setting it and verify it's being set to 6 or 7 to grab at 640x480...but it's still missing the edges...

public async Task Start(CancellationToken ct) {
            MMALCameraConfig.VideoResolution = Resolution.As03MPixel;
            var sensorMode = MMALSensorMode.Mode0;
            switch(camMode) {
                case 1:
                    sensorMode = MMALSensorMode.Mode1;
                    break;
                case 2:
                    sensorMode = MMALSensorMode.Mode2;
                    break;
                case 3:
                    sensorMode = MMALSensorMode.Mode3;
                    break;
                case 4:
                    sensorMode = MMALSensorMode.Mode4;
                    break;
                case 5:
                    sensorMode = MMALSensorMode.Mode5;
                    break;
                case 6:
                    sensorMode = MMALSensorMode.Mode6;
                    break;
                case 7:
                    sensorMode = MMALSensorMode.Mode7;
                    break;
            }
            MMALCameraConfig.VideoFramerate = new MMAL_RATIONAL_T(90, 60);
            MMALCameraConfig.SensorMode = sensorMode;

            using (var vidCaptureHandler = new EmguInMemoryCaptureHandler())
            using (var splitter = new MMALSplitterComponent())
            using (var renderer = new MMALNullSinkComponent()) {
                cam.ConfigureCameraSettings();
           .......

[d8ahazard]

And if the above code is a little odd or confusing, I've now also added bits so that I can specify the image dimensions and sensor type via config. (I've also removed the setter for the framerate)

I've tried every logical combination of sensor mode and scale division I can think of, and the result is always cut off.

Am I missing something? If I want the "full image", but the resolution could be 1/2 or 1/4 the size of full and work fine for my purposes - what should I be setting the sensor mode and resolution to? (Higher FPS is better for me)

[techyian]

I will have a look into it this evening and get back to you :)

[d8ahazard]

I will have a look into it this evening and get back to you :)

Brilliant, thank you. Just for your reference, here's the code I'm currently using to init the camera... Cam width is currently 1296, cam height is 972, and trying with a mode of 6, which I see in the code is listed as being "variable". I've legit tried with every camera mode though...6 is just my current setting.

 public async Task Start(CancellationToken ct) {
            MMALCameraConfig.VideoResolution = new Resolution(capWidth, capHeight);
            var sensorMode = MMALSensorMode.Mode0;
            switch(camMode) {
                case 1:
                    sensorMode = MMALSensorMode.Mode1;
                    break;
                case 2:
                    sensorMode = MMALSensorMode.Mode2;
                    break;
                case 3:
                    sensorMode = MMALSensorMode.Mode3;
                    break;
                case 4:
                    sensorMode = MMALSensorMode.Mode4;
                    break;
                case 5:
                    sensorMode = MMALSensorMode.Mode5;
                    break;
                case 6:
                    sensorMode = MMALSensorMode.Mode6;
                    break;
                case 7:
                    sensorMode = MMALSensorMode.Mode7;
                    break;
            }
            MMALCameraConfig.SensorMode = sensorMode;

            using (var vidCaptureHandler = new EmguInMemoryCaptureHandler())
            using (var splitter = new MMALSplitterComponent())
            using (var renderer = new MMALNullSinkComponent()) {
                cam.ConfigureCameraSettings();
                LogUtil.Write("Cam mode is " + MMALCameraConfig.SensorMode);
                // Register to the event.
                vidCaptureHandler.MyEmguEvent += OnEmguEventCallback;

                // We are instructing the splitter to do a format conversion to BGR24.
                var splitterPortConfig = new MMALPortConfig(MMALEncoding.BGR24, MMALEncoding.BGR24, 0, 0, null);

                // By default in MMALSharp, the Video port outputs using proprietary communication (Opaque) with a YUV420 pixel format.
                // Changes to this are done via MMALCameraConfig.VideoEncoding and MMALCameraConfig.VideoSubformat.                
                splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);

                // We then use the splitter config object we constructed earlier. We then tell this output port to use our capture handler to record data.
                splitter.ConfigureOutputPort<SplitterVideoPort>(0, splitterPortConfig, vidCaptureHandler);

                cam.Camera.PreviewPort.ConnectTo(renderer);
                cam.Camera.VideoPort.ConnectTo(splitter);

                // Camera warm up time
                await Task.Delay(2000).ConfigureAwait(false);
                await cam.ProcessAsync(cam.Camera.VideoPort, ct);
            }
        }

[techyian]

I've just had a look using both the OV5647 and IMX219 camera modules and it seems to be working as I'd expect it to.

To replicate Raspivid using the resolution 1296x972 using a framerate between 1-42fps, this should automatically select sensor mode 4 which has a full field of view. The config settings for this would be:

MMALCameraConfig.VideoResolution = new Resolution(1296, 972);
MMALCameraConfig.VideoFramerate = new MMAL_RATIONAL_T(40, 1); // This represents a fraction. The "num" parameter is the framerate divided by 1 to give you 40.
MMALCameraConfig.SensorMode = MMALSensorMode.Mode4; // This forces mode 4. Mode 0 (automatic) should figure out we want sensor mode 4 anyway based on the resolution and framerate.

There may be some slight confusion around the sensor mode and why you may want to force it (I may not have explained it as well as I could have). Using your camera module as an example, say you wanted to record at 1920x1080 but still wanted full field of view, you could set your framerate to match the constraint of mode 2 (1-15fps) and force that sensor mode. This would output video at 2592x1944 but the Pi's GPU would resize it for you to 1920x1080. Does that make any more sense?

[d8ahazard]

Hmmm. So, using the example above, this is the resulting image. Still cut off on the sides?

So, just so I'm clear, what I'm doing is breaking the captured video out into an edged version, finding the biggest 4-sided polygon, and then warping that out into the screen image.

Then I chop that up into a grid and figure out the average color per sector per frame. The edged image I'm already resizing to ~600x400, so overall resolution doesn't matter. I'm just aiming for a high frame rate and the full image...

[techyian]

Could you provide the raspivid command you're using so then I can do a comparison locally?

[techyian]

Also, are you able to achieve the scaling you're looking for by taking a picture? I'd be interested to find out whether it's isolated to video recording only?

The config settings you'll need to apply are:

MMALCameraConfig.StillResolution
MMALCameraConfig.StillFramerate

[d8ahazard]

I'm literally just doing "raspivid -o filename.h264 -t 3". Same for stills, I'm not specifying anything special. Just the command and the file name.

On Tue, Mar 3, 2020, 4:13 AM Ian Auty notifications@github.com wrote:

Could you provide the raspivid command you're using so then I can do a comparison locally?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/techyian/MMALSharp/issues/130?email_source=notifications&email_token=AAMO4NB5IYRNMXHTUARPKTTRFTJ5LA5CNFSM4K7FYE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENS4E5I#issuecomment-593871477, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NFBYTBGDHPKJKJUSB3RFTJ5LANCNFSM4K7FYE3A .

[d8ahazard]

And yes, scaling is also achieved when I take a still image.

Edit: Misunderstood this question earlier, will test as noted below...

On Tue, Mar 3, 2020, 6:55 AM Ben K. d8ahazard@gmail.com wrote:

I'm literally just doing "raspivid -o filename.h264 -t 3". Same for stills, I'm not specifying anything special. Just the command and the file name.

On Tue, Mar 3, 2020, 4:13 AM Ian Auty notifications@github.com wrote:

Could you provide the raspivid command you're using so then I can do a comparison locally?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/techyian/MMALSharp/issues/130?email_source=notifications&email_token=AAMO4NB5IYRNMXHTUARPKTTRFTJ5LA5CNFSM4K7FYE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENS4E5I#issuecomment-593871477, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NFBYTBGDHPKJKJUSB3RFTJ5LANCNFSM4K7FYE3A .

[d8ahazard]

raspivid -md 4 -t 3 -o t4.h264:

raspivid -md 6 -t 3 -o t5.h264:

Will try to write something to try taking a picture once...

[d8ahazard]

OK, added a function to snap a picture before initializing video. Using the set dimensions of 1296x972 and not setting any still framerate, it takes a normal picture:

But, the video is still cut off...

Is it something I'm doing after the video is being grabbed? Here's the full file I'm using to read the video data. I did try editing the splitter configs so they also used the same inputs, but that had no effect.

https://github.com/d8ahazard/HueDream/blob/DreamVision/Models/DreamGrab/PiVideoStream.cs

[techyian]

I've managed to reproduce this now and can see the differences between raspivid and Still image/Video capture in MMALSharp. I'll try and get to the bottom as to why this is happening.

[d8ahazard]

That's so great to hear. I was starting to think I was losing my mind.

Let me know if I can provide any other information or test anything.

On Wed, Mar 4, 2020 at 12:37 AM Ian Auty notifications@github.com wrote:

I've managed to reproduce this now and can see the differences between raspivid and Still image/Video capture in MMALSharp. I'll try and get to the bottom as to why this is happening.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/techyian/MMALSharp/issues/130?email_source=notifications&email_token=AAMO4NGPDJEF643GQBE75HDRFXZKLA5CNFSM4K7FYE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENWRVCQ#issuecomment-594352778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NEJFBGI3FBW43UGRNDRFXZKLANCNFSM4K7FYE3A .

[d8ahazard]

How goes it? :D

On Wed, Mar 4, 2020 at 7:46 AM Ben K. d8ahazard@gmail.com wrote:

That's so great to hear. I was starting to think I was losing my mind.

Let me know if I can provide any other information or test anything.

On Wed, Mar 4, 2020 at 12:37 AM Ian Auty notifications@github.com wrote:

I've managed to reproduce this now and can see the differences between raspivid and Still image/Video capture in MMALSharp. I'll try and get to the bottom as to why this is happening.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/techyian/MMALSharp/issues/130?email_source=notifications&email_token=AAMO4NGPDJEF643GQBE75HDRFXZKLA5CNFSM4K7FYE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENWRVCQ#issuecomment-594352778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NEJFBGI3FBW43UGRNDRFXZKLANCNFSM4K7FYE3A .

[techyian]

No good news yet unfortunately! I've tried a number of things so far. I find it interesting that the still image capture works properly but not video. If I don't get anywhere I'll raise a thread on the Pi forums for some official support.

[d8ahazard]

Sounds good, thanks for the update. Just for my own curiosity, where in the code are you setting the video mode?

On Fri, Mar 6, 2020, 12:26 PM Ian Auty notifications@github.com wrote:

No good news yet unfortunately! I've tried a number of things so far. I find it interesting that the still image capture works properly but not video. If I don't get anywhere I'll raise a thread on the Pi forums for some official support.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/techyian/MMALSharp/issues/130?email_source=notifications&email_token=AAMO4NFXSKWXOHAACO4SWJ3RGE55FA5CNFSM4K7FYE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCLBYA#issuecomment-595898592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NCWNCD2YV7TTMMYQBTRGE55FANCNFSM4K7FYE3A .

[techyian]

I've raised a ticket on the Pi forums to hopefully get an idea of where to be looking to fix this. I felt that it may have something to do with the sensor mode not being set correctly, but switching between mode 1 and 4 does appear to limit the Field of View considerably which I'd expect it to.

The main camera setup code is done here. Ports are then configured here. I've noticed a few differences in the way I set the camera up compared to raspivid however I've amended them locally with no difference seen. There seems to be some cropping appearing in the pipeline somewhere but I've not managed to track where yet. I'll keep you posted on progress.

[techyian]

Hello,

I've happened to come across this post written by 6by9 on the Pi forums. He has mentioned that the Video Stabilisation feature can cause the ISP to crop the video.

Call MMALCameraConfig.VideoStabilisation = false; prior to running ConfigureCameraSettings() and you should be good to go. It's fixed it for me locally. I felt that having it enabled by default would be a positive thing!

Let me know if there's anything else I can help you with.

[techyian]

I've raised #135 to turn this off by default.

[d8ahazard]

Brilliant! That fixed it! Now I'm off to sort out issues in my own code.

Thanks so much for the help. :D