search

raspberrypi / firmware

This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
5.15k stars 1.68k forks source link

MMAL: ISP: Is there a way to change the demosaic filter values? #880

Closed Terminus-IMRC closed 6 years ago

Terminus-IMRC commented 7 years ago

I'm thankful to you Pi Towers for enabling users to use the ISP for raw image demosaicing via MMAL. It is faster than CPU. I really appreciate it!

However, I have a question: Is there a way to change the demosaic filter values? The default filter is fixed, and borrowing values from other color components. So the output image is mostly greenish and it is difficult to collate colors of images.

Thanks.

6by9 commented 6 years ago

I'm afraid that there are no external controls possible for the demosaic block itself. It looks like the filter is fixed by the hardware, and the other parameters cause me to hesitate over IP release issues (it's Broadcom's IP and not Pi Towers').

What I suspect you really want to tweak is the colour conversion matrix (CCM) immediately following the demosaic - that defaults to the values used on the IMX219, which obviously isn't suitable in all situations. A parameter got added to control that a month or so back for other reasons - MMAL_PARAMETER_CUSTOM_CCM under MMAL, or OMX_IndexParamCustomCcm for IL. Under MMAL the values are rationals which do get converted to 16Q16 in IL land. I'd need to double check, but I believe that is also the CCM that is written to the Makernote when used with the firmware camera stack. Default within vc.ril.isp is

 2.10  -0.92  -0.18
-0.37   1.52  -0.14
 0.12  -0.98   1.87

(This is also active when feeding YUV or RGB in, and needs to convert the input to accordingly. Probably best to leave alone in those situations as the pipe is configured differently in each case. At some point I'll get it all nice and consistent, but not yet).

The other bit that needs to be exposed is the white balance gains. They get applied to the red and blue channels post lens shading but prior to demosaic. Digital gain also falls into that block I think. You could fake it at the moment by altering the lens shading table, but that's a hack rather than doing it properly. All details from https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=190586 for lens shading should also apply to the vc.ril.isp component, although I haven't done as much testing there.

Terminus-IMRC commented 6 years ago

Thanks! I tried to apply custom CCM to vc.ril.isp->input[0] and succeeded to do! And the CCMs can be differ for multiple ISP instances! Wow!

It's a little sad that the demosaic filter cannot be changed. Could you tell me the filter parameters (or the size of the filter) so I can tweak the values on CPU side?

And thank you for telling me the internal pipeline order!

6by9 commented 6 years ago

Everything can be different for different ISP instances. The whole ISP is tile based and the control software has a simple scheduling algorithm for which tiles to do next. The full pipe config is attached to the tiles rather than being a global.

Sorry, I really can't give details of the demosaic algorithm as it is not just a simple static filter. It's a 5x5 context, but that's about all I can say. The documentation is 3 pages of fairly terse maths that it'd take me a good couple of hours to understand!

A slightly more complete list of the pipeline stages is documented at http://picamera.readthedocs.io/en/latest/fov.html#background-processes, although that omits the CCM after demosaic, and doesn't mention the low res resizer and output. I've been busy on other things recently but do intend to get most of those exposed, at least with a fairly coarse control. The other bit that isn't mentioned is statistics (Bayer input only), and hopefully I can add those as another output so that 3A algorithms can be added efficiently.

Terminus-IMRC commented 6 years ago

Thanks for a lot of information!

BTW, are issues about vc.ril.rawcam still welcome? Should we now migrate to V4L2 Unicam driver? (I have an issue about vc.ril.rawcam that makes the entire system down.)

6by9 commented 6 years ago

BTW, are issues about vc.ril.rawcam still welcome? Should we now migrate to V4L2 Unicam driver? (I have an issue about vc.ril.rawcam that makes the entire system down.)

If it's that killing the app whilst streaming kills the GPU, then it's known. It's a double free that I haven't tracked down. If you add start_debug=1 to config.txt, then sudo vcdbg log assert will dump out all asserts and generally include an error in a blockpool IIRC.

Report it, and I'll look at it at some point.

Terminus-IMRC commented 6 years ago

Thank you! I'll raise another issue.

Thank you again!

Terminus-IMRC commented 6 years ago

I said there was a bug in rawcam, but it turned out that it was my mistake (invalid values are set to IMX219's width and height registers). Sorry for that. Now my library is working! Thank you for your great frameworks and supports!