optimizing PhotoneoYCoCg420 pixel format conversion

[bmegli]

As identified in:

• https://github.com/Extend-Robotics/camera_aravis/issues/10#issuecomment-1653587175

After enabling internal benchmark:

• color conversion + publishing takes 10-35 ms
• most of this time is color conversion
• this is terrible result for sub 1 megapixel image

[bmegli]

The current code was taken from Photoneo example:

• https://github.com/photoneo-3d/photoneo-cpp-examples/blob/057428aa15b54234c385712e88e92ddaae90246c/GigEV/aravis/common/YCoCg.h#L66

[bmegli]

In the example code we can read:

/**
 * Utility class providing conversion function for images.
 * Provides conversion
 *     BGR  <--> YCoCg 4:2:0,
 * where the chromatic channels Co, Cg are subsampled in half resolution.
 * The 2x2 plaquette of the YCoCg format consists of:
 *   (0, 0): Y (10 bits), 1st half of Co (6 bits)
 *   (1, 0): Y (10 bits), 2nd half of Co (6 bits)
 *   (0, 1): Y (10 bits), 1st half of Cg (6 bits)
 *   (1, 1): Y (10 bits), 2nd half of Cg (6 bits)
 *
 * Black pixels are treated specially in order to prevent artifacts in images containing valid pixels only in a subregion.
 *
 * NOTE: The YCoCg format used here differs from the reference by:
 *        - a factor 2 in the both chromatic channels Co, Cg,
 *        - an offset equal to the saturation value, which is added to Co and Cg to prevent negative values.
 *
 * @see https://en.wikipedia.org/wiki/YCoCg

    // Subsampling of the chromatic channels Co, Cg.
    // The subsampling is meant both in horizontal and vertical direction.
    static const int chromaSubsampling = 2;

    // Pixel depth of B, G, R, and Y channels. Chromatic channels Co and Cg have an additional bit.
    static const int pixelDepth = 10;

 */

[bmegli]

Examining the code:

• the pixel format doesn't look to me as YCoCg

It looks as reversible YCoCg-R

• with 10 bit depth for Y
• 11 bit depth for Co, Cg

YCoCg-R was introduced in this Microsoft paper:

• https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/2008_ColorTransforms_MalvarSullivanSrinivasan.pdf

[bmegli]

YCoCg-R -> RGB

The 10 bit depth variant is also described in the VESA standard

• https://glenwing.github.io/docs/VESA-DSC-1.2.pdf

"7.7 Color Space Conversion" has algorithm for converting back to RGB

[bmegli]

RGB -> YCoCg-R

Looking at Photoneo comments

/*
NOTE: The YCoCg format used here differs from the reference by:
 *        - a factor 2 in the both chromatic channels Co, Cg,
 *        - an offset equal to the saturation value, which is added to Co and Cg to prevent negative values.
 */

This also looks as YCoCg-R defined in VESA

"6.1 Color Space Conversion"

[bmegli]

Original

• up to 30 ms

[bmegli]

Quick and dirty optimization trial

• up to 11 ms

[bmegli]

Asked Photoneo

• https://github.com/photoneo-3d/photoneo-cpp-examples/issues/4

[bmegli]

Original vs Optimized

We need to fix the clock or check in performance mode to eliminate variability of test result.

Retesting in performance mode

• ~7 ms Original vs ~3 ms Optimized

Systems don't run on maxed clocks so the expected behavior in production is closer to previous tests

Original

Optimized

[bmegli]

That's good enough for now.

In case we need faster processing

• it can be easily implemented with SIMD
  • 4x pixels at a time should be straightforward
  • roughly matches current code structure

[bmegli]

One more test

• optmized RGB8
• vs optimized BGRA8

The second one maps directly to VAAPI bgr0 so avoids another color conversion later (not in the scope of this repository)

The time measured here

• includes H.264 hardware encoding
• there is not much difference between implementations in terms of time
• BGRA8 uses a bit less CPU
  • visible when watching both, a few percent more lightweight

RGB8

BGRA8