image quality issues with mmal

[alligitor]

0. Reviewed guide and contributing documents? (Yes/No): yes
1. version [x.y.z, hash, other]: 4.1.1
2. installed as a package or compiled from sources [deb, rpm, git, other]: git
3. standalone or part of third party [motion, MotionEyeOS, other]: standalone
4. video stream source [V4L (card or USB), net cam (mjpeg, rtsp, other), mmal]: mmal
5. hardware [x86, ARM, other]: Rasp Pi 3 arm71
6. operating system [16.04, Stretch, etc, FreeBSD, other]: Raspian

I've been trying to make a motion security camera with RaspPi and the Raspberry pi NoIR camera for a while and can't get it to work right. I've done a lot of experimentation with various options and hoping somebody can point in the right direction.

there are 2 problems 1- the night vision/exposure is poor with motion. to show a sample, I'm including 3 pictures that where taken in a dark utility room, with 2 IR light sources. 1 of the pictures is taken with a Foscam night vision camera to put this in as a comparison. The other two pictures are from the same exact Raspberry Pi, one taken with motion, other with raspistill. the image with foscam is very vivid. raspistil's picture is also pretty bright. picture from motion is not exposed properly.

2- the picture out of motion doesn't use the whole view of the camera. as it can be seen from the raspistill picture, the camera can see a rather wide area but motion only shows a fraction of the view. I've tried changing the -h and -w, but instead of seeing the whole view at different resolutions, the field of view changes depending on the resolution.

the relevant parameters in motion.conf are:

mmalcam_name  vc.ril.camera
mmalcam_control_params -ex nightpreview -w 1920 -h 1080 -rot 90
rotate 0
width 1920
height 1080

the raspistill picture was taken using the same parameters, passed to raspistill raspistill -ex nightpreview -w 1920 -h 1080 -rot 90 -o test.jpg

I've also tried: mmalcam_name vc.ril.camera mmalcam_control_params -ex nightpreview -w 1920 -h 1080 -rot 90 -ISO 1600 -roi 0,0,1,1

the foscam picture is just to have another camera as reference. --------- here are the pictures ---------------

[genius3000]

I can't test the mmalcam functionality myself so I could only take wild guesses without more information. Try starting motion with log level INF (7) and logging to a file so you can upload it to a gist, take a snapshot and stop it. I see the images both scale down some, with motion being slightly more but it's also "zooming in" from the looks of it.

[alligitor]

gist created here; https://gist.github.com/alligitor/7dcc68e4dc023ceb771a46b54f08f7db

[genius3000]

I don't see any clues as to the image size thing, other than maybe it's re-scaling for the quality of 75% (could try bumping that to 100).
I do see that auto_brightness is off but, brightness is set to "128" (which is normal when config'ed with "0" and auto_brightness on). Maybe try setting brightness low and see if the auto_brightness is messing it up. I'm not even sure if those camera options affect the mmalcam though.
That's all the ideas I have, hopefully someone else has further insight for you.

[alligitor]

genius3000, thanks for looking.
how familiar are you with structure of Motion? I'm looking for a high level view of how pieces are put together. for example, the flow from getting the image from device to motion detection to writing to disk.

another part is about setting up the camera for image capture. where do parameters get set, specifically ones related to exposure.

Does such thing exist?

[Mr-Dave]

The short version is that it starts with the motion.c mlp_functions that run the motion loop. The loop does a start, capture, detect, process actions, do it again. The capture step uses the video_common functions which then branch out to either a network camera, v4l2 device or mmal device. All of those routines then have a start, next, cleanup set of functions.

For the MMAL method of using the pi cam, the place to look is in the mmalcam.c module. Then as needed, look into the rapicamcontrol routines which are direct copies of what is provided by the Raspberry PI userland.

I assume of course that you have tried already the parameters of exposure that provided via the change I pushed when using the pi cam via the V4L2 device method as well as tried the exposure parameter that is built into the MMAL.

[alligitor]

Mr-Dave, I read somewhere that the parameters listed in motion.conf for "mmalcam_control_params" are pretty much the same as what raspistill uses. So tried a number of different option under raspistill and motion to compare. For example: mmalcam_control_params -ex night -w 1920 -h 1080 -rot 90 -ISO 1600 -roi 0,0,1,1 In my experience, same parameters gave different results.

when you say: "parameters of exposure that provided via the change I pushed when using the pi cam via the V4L2 device method" are you talking about changes submitted as part of: https://github.com/Motion-Project/motion/commit/46b5f88140beb7e58f82660ac72eb0c2242ec04c

or something else.

[Mr-Dave]

Yes. Those are the parameters. Motion just passes those along to the mmal libs for processing and the mmal libs sends Motion back a image. There is little to no interpretation / adjustment happening in Motion. This line starts by setting the camera to the defaults, then it copies the user parameters to the mmal structure which is passed into the libs.

Once the cam is started, there is just a memcpy to get the image. Once the image is in the image_norm, we rotate it, apply any masks etc none of which did I see being applicable in your config.

To test, you may want to add a line right after the memcpy to write the image to a file. This way you'd know whether it is provided by the camera that way or whether Motion somehow changed it after capture.

With respect to the V4l2, you are correct. It was that commit in which I added functionality to Motion to permit the changing of all camera parameters. (Description of how is in the git version of the guide

EDIT: One additional note. I put in the -ex night for the parameter for my "regular" pi cam and noticed that it took about 5 seconds of capture for MMAL libs of the camera to settle down to a good picture. Have you let Motion (and the mmal libs) run for a bit to see if it adjusts the exposure to the ambient light level?

[alligitor]

I have tried waiting for minutes. doesn't make a difference. Also, I have two RPv3 + NoIR cameras. both behave the same.

I added in mmalcam_start in 3 spaces just to see how the parameters I set in motion.conf get passed down to the mmcal library.

here's what I got. looks as expected.

**mmalcam parameters after raspicamcontrol_set_defaults**
Sharpness 0, Contrast 0, Brightness 50
Saturation 0, ISO 0, Video Stabilisation No, Exposure compensation 0
Exposure Mode 'auto', AWB Mode 'auto', Image Effect 'none'
Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128
Rotation 0, hflip No, vflip No
ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000

**mmalcam parameters after parse_camera_control_params**
Sharpness 0, Contrast 0, Brightness 50
Saturation 0, ISO 1600, Video Stabilisation No, Exposure compensation 0
Exposure Mode 'night', AWB Mode 'auto', Image Effect 'none'
Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128
Rotation 0, hflip No, vflip No
ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000

[1:ml1:RP-Camera09] [NTC] [VID] create_camera_component: MMAL camera component created

**mmalcam parameters after create_camera_component**
Sharpness 0, Contrast 0, Brightness 50
Saturation 0, ISO 1600, Video Stabilisation No, Exposure compensation 0
Exposure Mode 'night', AWB Mode 'auto', Image Effect 'none'
Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128
Rotation 0, hflip No, vflip No
ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000

[alligitor]

Excellent idea about dumping the image after mmalcam_next gets a frame. I put a call to dump the buffer: put_picture(cnt, "test_pic.jpg", img_data->image_norm, FTYPE_IMAGE);

Here's what I found:

Given these options in motion.conf: mmalcam_name vc.ril.camera mmalcam_control_params -ex night --width 1920 --height 1080 -ISO 1600 -ss 11904 and: width 1920 height 1080

The file, "test_pic.jpg", came out as a 1920x1080 and matched what is displayed on motion's output to my web browser.

Interestingly, I ran: raspistill -ex night --width 1920 --height 1080 -ISO 1600 -ss 11904 -o raspi.jpg

and it generated a file on disk that was 1920x1080, BUT, it had a larger view of the surrounding. So, seems like cropping isn't done by Motion. somehow mmal library returns a smaller view, than raspistill does!

However, the exposure is the same.

[Mr-Dave]

Thanks for the edit/update.

I did review the code for raspistill/raspivid on their github site. The only thing I can think of on the cropping is it could be that we need a VCOS_ALIGN_UP on the dimensions as they are assigned. (This really seems like a long shot however as the dimensions for your images are already aligned to 32 and 16.)

For the exposure, I could have missed something but it seems we are doing the same thing in Motion as they are doing in raspivid code. (The raspivid is the basis of the code in Motion rather than raspistill)

[alligitor]

I did a quick review of raspistill code too and based on my brief experience, looked like they do somethings a bit differently. They seem to be using callbacks for interacting. Is it possible that makes a difference? Also, there were 2 places that I say them re-writing some aprameter (shutter speed) saying that it needs to be rewritten. could that make a difference?

[Mr-Dave]

I am concentrating on the raspivid instead of raspistill since the Motion code aligns with raspivid. Can you test the raspivid and validate the exposure is still an issue.

Assuming that it is still an issue in raspivid, the following are my observations.

1. The Motion code is not setting up the preview port. Perhaps the noir camera needs that to set the exposures.
2. We don't have this call status = mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_CUSTOM_SENSOR_CONFIG, state->sensor_mode); within Motion and instead we are taking the default. The pi forums seem to indicate that changing the sensor_mode could help with the cropping. (The forum was indicating that there was cropping differences between raspistill vs raspivid and manually setting the sensor_mode to something other than the auto default helped).
3. Looking at the callback, I did not see that it would assist us. It seemed to be just logging the results rather than invoking new settings or methods.
4. The shutter speed seems to only be applicable when specified by the user. From your initial parameters, it didn't look like it was specified to raspistill and yet raspistill gave a decent image.
5. Thinking about the shutter speed, I noticed in the Motion parms a frame rate of 100 was specified. Can you try a low number such as 2 or 10? Note in the port section of our mmalcam.c code how that is being passed into the lib. Perhaps the lib can not handle such a high number.

Unfortunately, since I can not observe the issues with my camera, I can't test any of these.

[alligitor]

I'll try your suggestions.

meanwhile, re: "Unfortunately, since I can not observe the issues with my camera, I can't test any of these." If you have an interest to play with the NoIR camera personally, I could donate one. Let me know.

[Mr-Dave]

For item 1, I noticed that some of the raspi documents indicate that the preview port is required to set the exposure and white balance. I have pushed a raspi_preview branch to my development repo that I think implements the preview_port.

Can you pull that repo and branch and test it to see if it resolves the exposure issue?

[alligitor]

Tried it. it didn't make a difference for exposure but created a green bar at the top.

[Mr-Dave]

Hmpf...going backwards and running out of options.

Have you tried the other scenarios?

1. Try raspivid and good vids?
2. Lower framerates?
3. Maybe try to uncomment the line in 'set_preview_port_format' to execute the format->encoding = MMAL_ENCODING_I420; It was not observed in the raspivid code but it seemed strange to omit it.

On the donation front, generally I've observed that other people get weird when money / gifts get involved..."how come I didn't get any? I did this and that"...Which is why I've tried to stay away from that and only do this for stars on the project (hint hint). (The local store is currently out of stock anyway and I'd have to spend another $5 for another pi zero).

[alligitor]

on

2.Lower framerates?

I wouldn't have thought about this if you hadn't suggested it. I tried lower frame rates and it does make a difference. actually, makes too much of a difference; maybe a fluke but i tried a couple of times.

Lowering the frame rate from 100 to 10 made the picture look nice and in gray scale. Lowering the frame rate further give it a green tint. Pictures for frame rates 10 and 2 attached.

I'll run with framerate of 10 for a while to see what I get at night, day, etc.

IMO, Price wise, it's a bit cheaper to pair up a webcam with the Pi, but from size/compactness you can't beat the Pi Camera.
My local store sells these: https://www.raspberrypi.org/magpi/pi-zero-w-official-case/ for $5. $7 for RPZ-W + $5 case + $29 Camera + $8 uSD and you get a pretty nice little Motion detector.

So I think it's good for you guys (motion team) have these on hand for testing, troubleshooting, etc. I am in US. If you ever send me your info privately, I could send a setup your way.

I'll try the other two options later, but I think you nailed it with the frame rate.

1.Try raspivid and good vids?
3. Maybe try to uncomment the line in 'set_preview_port_format' to execute the format->encoding = MMAL_ENCODING_I420; It was not observed in the raspivid code but it seemed strange to omit it.

[Mr-Dave]

Thanks for the update. If you observe an upper limit on the frame rate, please report back and I can impose a limit in the code as it is assigns it to the port and and report it in the log to the user.

I'd guess the green hue may go away if you let it run for a while. This is what I was observing with the normal pi cam in low light.

The libs probably need a set number of frames to establish the exposure and the high frame rate was trying to stuff them into a fraction of a second vs a frame rate of two spreads them out over many seconds/minutes.

[alligitor]

Interesting experience. Here's the result of testing with different frame rates. As rates go up from 10, the pictures keeps getting slightly darker and darker. Perhaps the camera firmware honors the frame rate and sends under exposed frames. Interestingly, at 45 frames per second, the pictures also get cropped.

I wouldn't impose a hard limit in the code based on this experience. Maybe the limit is related to amount of ambient light.

Perhaps update the docs? warning message? Alternatively, I can open a "new issue" titled something like: "Frame rate affects picture quality with Raspberry Camera" and put the following pictures and let others comment too and we'll see what other people experience.

Here are 3 frames I captured at 25, 40 and 45.

[alligitor]

Nice job Mr-Dave. Thanks for the help