ImageBufferAlgo::demosaic (IBA::demosaic)

[ssh4net]

OpenImageIO already supports different demosaic algorithms via the libraw or OpenCV plugin.

Technically, as soon as libraw is already built and the library is present, it is possible to call libraw to demosaic routine passing binary blob to it. That should work fine for 8/12/16-bit inputs. In most cases, users are OK with that.

But if you need demosaic float imagery there are only OpenCV can help. The problem is that OpenCV as a meta library is probably even bigger than OIIO. And building OIIO with OpenCV only for demosaic is probably a huge overhead.

So, the idea is to implement a lightweight Demosaic function as a part of ImageBufferAlgo with the support of all bit depths OIIO.

For start it can be a simple Linear Demosaic. Later it can be extended by adding other open source algorithms.

[lgritz]

There isn't really an "OpenCV plugin", just really simple functions to convert between a cv::Mat and an ImageBuf. (In fact, #4363 even makes them just inline function utilities, eliminating OpenCV as a direct dependency of libOpenImageIO).

If libraw (a) exposes the demosaicing from/to a buffer separately from their reading whole raw files, so we can call that directly, and (b) we trust their implementation and don't want to do anything different, and (c) we're ok with IBA::demosaic() only working if the optional libraw dependency is found at build time, then we could implement IBA::demosic as basically a wrapper around the libraw functionality.

But if you think that it's better for us to control our own fate, so to speak, then maybe we just want a completely new, full implementation of demosaicing in IBA::demosaic without any reliance on libraw.

[ssh4net]

int LibRaw::open_bayer(unsigned char *data, unsigned datalen, ushort _raw_width, ushort _raw_height, ushort _left_margin, ushort _top_margin, ushort _right_margin, ushort _bottom_margin, unsigned char procflags, unsigned char bayer_pattern, unsigned unused_bits, unsigned otherflags, unsigned black_level)
Parameters:

data, datalen - buffer passed
_raw_width/_raw_height/*margin - image size and margins
procflags:
for 10-bit format:
1: "4 pixels in 5 bytes" packing is used
0: "6 pixels in 8 bytes" packing is used
for 16-bit format:
1: Big-endian data
bayer_pattern: one of LIBRAW_OPENBAYER_RGGB,LIBRAW_OPENBAYER_BGGR, LIBRAW_OPENBAYER_GRBG,LIBRAW_OPENBAYER_GBRG
unused_bits: count of upper zero bits
otherflags:
Bit 1 - filter (average neighbors) for pixels with values of zero
Bits 2-4 - the orientation of the image (0=do not rotate, 3=180, 5=90CCW, 6=90CW)
black_level: file black level (it also may be specified via imgdata.params)
See samples/openbayer_sample.cpp for usage sample (note, this sample is 'sample only', suited for Kodak KAI-0340 sensor, you'll need change open_bayer() params for your data).

For integer pixels, that's probably a wise choice. But they do not support floats. The main advantage of libraw is that it can decodes a bunch of vendor-specific camera formats. A lot of commercial projects do a demosaic on it's own side and on GPU and use only decoding from libraw.

[lgritz]

Based on what you wrote above, it also looks like the libraw entry point wants the data encoded in a very specific way, which is probably related to the libraw decoding internals and does not resemble how data will be laid out in an ImageBuf. Therefore, trying to use that libraw function to and from IB will involve extra data translations in and out, which is really going to hurt performance and seems silly. It's probably better to write an IB-to-IB demosaicing function from scratch.

[antond-weta]

I've implemented a simple linear demosaicing, just to learn how to write an IBA.

4366

@ssh4net the code is functional, I've tested it on a bunch of images, in case you wanted to give it a go. There is still room for optimisation, but that is not a priority at this stage.

@lgritz, I've got a few questions I'd like answered before continuing the work on the actual algos:

• The input is always expected to be single channel? Is it ok to error if there are multiple channels, or should the code create a separate RGB triplet for each input channel?
• Is ROI input referred? If there is an initialised ROI provided, do we expect it to have 1 channel or 3 channels?
• If the input buffer is integer, there is a possibility of overflow while doing averaging. It maybe worth using a one-up sized intermediate type for calculations. I'm currently using OIIO_DISPATCH_COMMON_TYPES2() which doesn't provision for that. Is there a better way?

I've also put a simple test into imagebufalgo_test which converts a bunch of raw images with hardcoded filenames. That is for debugging, I will make a proper test with generated images later.

[ssh4net]

Wow! So fast!

I also thought to use own cpp file for demosaics code.

As soon as we will definitely implement more than one interpolation algorithm, that might be worth on start implement some switch for using algo. Maybe similar to IBA::blur, just made a single entry "linear".

As for single channel vs RGB.

When I thought to implement demosaic I hold in mind only single channel images. But your question remind me another use case.

Usually raw data is single channel. But people can decode raw sensor data using libraw/dcraw to RGB mosaiced images. And that representation can be more common for people not deeply in topic. Memory and speed wise maybe worth to have separate implementation for grayscale/RGB data?

Regarding bit depth. It's better to do an averaging and all computation in float. And explicitly cast result to source or target bit depth.

[antond-weta]

I've refactored the code a bit. There is now a base class BayerDemosaicing and 2 subclasses LinearBayerDemosaicing and MHCBayerDemosaicing. The latter implements the Malvar-He-Cutler algorithm (https://www.ipol.im/pub/art/2011/g_mhcd/article.pdf). It is just slightly slower than the simple linear one, but gives significantly better results.

I am actually not convinced that we should vendor multiple demosaicing methods. I would rather have a single robust algo for each sensor type.

Should we want to expose the base class in the public interfaces, it would be pretty easy for the users to implement additional algos on their side. All they'd need to do is to specify the kernel size as a template param and override 4 methods implementing convolution for the 4 photosite types.

[ssh4net]

Great!!

I thought about AHD (ADAPTIVE HOMOGENEITY-DIRECTED DEMOSAICING ALGORITHM) which is a default libraw algorithm and the one had a good balance in quality and speed.

I think a single function ImageBufAlgo::bayer_demosaic() with string_view method similar to IBA::unsharp_mask is more convenient than use per algorythm public function.

OIIO::ImageBufAlgo::unsharp_mask(const ImageBuf &src, string_view kernel = "gaussian", float width = 3.0f, float contrast = 1.0f, float threshold = 0.0f, roi = {}, int nthreads = 0)

Or you think that some algo might require additional parameters?