Experimentation

[VittorioAccomazzi]

identify the filtering approach which allows to perform a consistent and pleasing abstraction. Good source of information :

• Image Abstraction by Structure Adaptive Filtering by by Jan Eric Kyprianidis and Jürgen Döllner
• Image and Video Abstraction by Coherence-Enhancing Filtering by Jan Eric Kyprianidis and Henry Kang
• Real-time video abstraction by Holger Winnemoeller

[VittorioAccomazzi]

Gaussian Bilateral Filtering

Implemented the Gaussian bilateral filter which is mention by all the paper above. The implementation is straightforward, and tested on RGB, HSV and Lab colour spaces. Here is an example in Lab with:

• spatial sigma 6
• range sigma 12 Notice that the
• first image is 512x512
• second image 1024x768

[VittorioAccomazzi]

DoG for contours

Implemented the Difference of Gaussian for define the contours in the image. here are the results with:

• sigma1 = 2
• sigma2 = 1.6* sigma1 using the luminosity component of the Lab decomposition:

[VittorioAccomazzi]

Color Quantization

Implemented a simple colour quantization, taken from Real-time video abstraction but without the gradient terms, since it seems smooth enough. Here is the quantization at 12 values, and it is done on the luminosity component of the Lab decomposition only. the image on the left is smoothed with a bilateral Gaussian filter and the DoG has been overlaid:

[VittorioAccomazzi]

Tensor Filter

As outlined in Image and Video Abstraction by Coherence-Enhancing Filtering the image structure tensor can capture very effectively the variation of the image, and so relevant details.

Simple Approach

Implemented a simple 'brute force' approach : simple smoothing on the along the direction of the minor eigenvector:

• On the left the image smoothed with 2 iteration of gaussian 1 sigma each, on the right the direction of the minor eigenvector
• on the left the image smoothed with 3 iteration of f gaussian 1 sigma each and on the right the original image: using this simple approach and interleaving the tensor filter with the bilateral gaussian improves to some abstraction with faces (look for instance at the hair): (left image tensor follow by a bilateral Gaussian, right bilateral gaussian only) however the landscape seems to be worst, since it enhance unrelevant details:

the code used is the following:

const numSteps = 5
const scaleStd = 2
const rangeStd = 2
const scaleDog = 3

export default class Painter {

....

    next() : SeqCanvas | null {
        let outCanvas = null

        if( this.step < numSteps) {
            [this.lImage, this.aImage, this.bImage] = TensorFilter.Run(this.lImage!, this.aImage!, this.bImage!,1) as [ ImageFloat32, ImageFloat32, ImageFloat32]
            [this.lImage, this.aImage, this.bImage] = TensorFilter.Run(this.lImage!, this.aImage!, this.bImage!,1) as [ ImageFloat32, ImageFloat32, ImageFloat32]
            [this.lImage, this.aImage, this.bImage] = BilateralFilter.Run(this.lImage!, this.aImage!, this.bImage!, scaleStd, rangeStd) as [ ImageFloat32, ImageFloat32, ImageFloat32]
            outCanvas = CanvasUtil.fromLab(this.lImage, this.aImage, this.bImage)
        } else if ( this.step === numSteps ){
            this.luminosityQuantization()
            outCanvas = CanvasUtil.fromLab(this.lImage!, this.aImage!, this.bImage!)
        }

        .....
    }
.....

}

[VittorioAccomazzi]

Painter Improvement

Adding tensor filter

The tensor filter does improves the abstraction of the hair, even if the implementation is still very simplistic, as shown in the images above. Using the code posted above.

Remove the border line

instead of adding the dark border line, added the original image with the same weights, the code is the following:

    private overlayDog( inCanvas : SeqCanvas) {
        let dogImage = this.dImage!
        let width = dogImage.width
        let height= dogImage.height
        let nPixels= width*height
        let dogPixels= dogImage.imagePixels
        let orgPixels= this.oImage.data // original image
        let ptr=0
        for( let p=0; p<nPixels; p++ ){
            let weight = dogPixels[p]
            inCanvas.data[ptr] = weight * inCanvas.data[ptr]+(1-weight)*orgPixels[ptr++]
            inCanvas.data[ptr] = weight * inCanvas.data[ptr]+(1-weight)*orgPixels[ptr++]
            inCanvas.data[ptr] = weight * inCanvas.data[ptr]+(1-weight)*orgPixels[ptr++]
        }

[VittorioAccomazzi]

Tensor Shock Filter

Implemented the LoG along the major eigenvector direction, as described in Image and Video Abstraction by Coherence-Enhancing Filtering with the following variation:

• The processing uses a nearest neighbour interpolation.
• LoG kernel precomputed to gain speed.

Tensor LoG

Here below are the image obtained from the LoG filter, which indeed edges. Sigma is 1.5, as suggested in the article

Tensor Shock

Here is the shock filter which uses the log above. The computation is done with the L component in Lab space as follow:

       let [lImg, aImg, bImg] = CanvasUtils.toLab(toSeqCanvas(canvas))

        GaussianFilter.Run(lImg,1)
        let tf = TensorGenerator.Run(lImg, aImg, bImg, 1) 
        let flt= TensorShock.Run(lImg, tf, 2, 0.01, 2) as ImageFloat32

[VittorioAccomazzi]

Painter Variation

Changing the painter algorithm entirely, as follow:

• tensor directional smooth with sigma 1.5
• tensor shock filter with sigma 1.5

The result provides a nice level of abstraction in several images, however not entirely consistent. Notice that:

• it is important that the sigma values in tensor smoothing and tensor shock to be the same and between 1.0 and 2.0
• intensity quantization is removed
• bilateral filter is removed. This type of algorithm tend to create some fix pattern on hairs and trees. The code is the following

      const numSteps = 6
      ......
      if( this.step < numSteps) {
          [this.lImage, this.aImage, this.bImage] = TensorDirSmoothing.Run(this.lImage!, this.aImage!, this.bImage!,1.5,tRelax) as [ ImageFloat32, ImageFloat32, ImageFloat32]
          let tf = TensorGenerator.Run(this.lImage!, this.aImage!, this.bImage!, 1,tRelax)
          this.lImage = TensorShock.Run(this.lImage!, tf, 1.5, 5, 1) as ImageFloat32
          outCanvas = CanvasUtil.fromLab(this.lImage, this.aImage, this.bImage)
      } 

  The pictures used above are the ones looking the worst:

And some of the artefact created are consistent with what I see here. however some of the images with fur head shots are looking quite good, but quite inconsistent:

[VittorioAccomazzi]

Final Decision : Tensor and Bilateral Painters

Based on this experimentation the decision is to generate to painters:

• A tensor based Painter as tested in here. It works well in several images, but in some just generate artefacts, which are clearly annoying.
• A bilateral filter tensor as tested in here. It works ore consistently then the tensor, however the level of abstraction is not as good.

Here are comparisons, on the left the tensor image, on the right the bilateral