[QUESTION] SKImageFilter detailed documentation?

[ziriax]

I'm trying to get a deep understanding of Skia's powerful image filter pipeline, but I'm having some trouble.

I read the nice SkiaSharp image filter introduction, but I miss many details.

For example, what is coordinate system used in filters? The Skia source code has the following comment:

/**
 *  Base class for image filters. If one is installed in the paint, then all drawing occurs as
 *  usual, but it is as if the drawing happened into an offscreen (before the xfermode is applied).
 *  This offscreen bitmap will then be handed to the imagefilter, who in turn creates a new bitmap
 *  which is what will finally be drawn to the device (using the original xfermode).
 *
 *  The local space of image filters matches the local space of the drawn geometry. For instance if
 *  there is rotation on the canvas, the blur will be computed along those rotated axes and not in
 *  the device space. In order to achieve this result, the actual drawing of the geometry may happen
 *  in an unrotated coordinate system so that the filtered image can be computed more easily, and
 *  then it will be post transformed to match what would have been produced if the geometry were
 *  drawn with the total canvas matrix to begin with.
 */

This helps a bit, but the exact definition of the "drawn geometry" is still obscure.

For example, when drawing a cropped image, the "drawn geometry" seems to be the cropped image, not the original image before cropping.

And indeed, the canvas transformation (device space) itself seems to be ignored.

Other behavior baffles me, for example, the following code fills the whole canvas with an image, not just a single pixel:

var paint = new SKPaint
{
    Color = 0xFFFFFFFF,
    Style = SKPaintStyle.Fill,
    ImageFilter = SKImageFilter.CreateImage(image)
};

canvas.DrawRect(0, 0, 1, 1, paint);

Further exploration will most likely raise my more questions ;-)

I think a great tool to learn Skia image filters would be a graphical pipeline with nodes (e.g. a very simple version of Blackmagic Design Fusion VFX), side-by-side with the source code. Maybe such a tool already exists?

In general, any documentation about Skia image filters would be welcome 😃

Thanks, Peter

PS: Maybe it's better to ask these questions in the Google Skia mailing list?

[wieslawsoltes]

More documentation would be really appreciated.

I am currently doing SVG Filter Effects and using extensively SKImageFilter in my implementation (which is sadly not yet correct on working as expected), maybe you can take some inspiration from that.

https://github.com/wieslawsoltes/Svg.Skia/blob/a362239968ea7201d2e376c8c16471069c1bfe54/src/Svg.Skia/SvgFilterskExtensions.cs#L12

The image filter topic is complex as you can see in my sample code and as you can see the specification for filter effects: https://www.w3.org/TR/SVG11/filters.html

[ziriax]

That is very interesting, thanks for the info.

Actually about 3 years ago I had to abandon using SVG filters for my customer's web image processing, because none of the browsers at the time implemented it correctly. So this must be very hard to get right.

[wieslawsoltes]

The Skia has excellent building blocks for implementing SVG filter effects. In SVG you basically render all filter effects to offscreen layer and then blend it into final canvas.

So basically what I do in my library (at least currently):

Step 0 - create paint with filter (you can chain filters too using input argument, you can also crop affected area)

https://github.com/wieslawsoltes/Svg.Skia/blob/a362239968ea7201d2e376c8c16471069c1bfe54/src/Svg.Skia/Drawables/Drawable.cs#L444

Step 1 - create layer

https://github.com/wieslawsoltes/Svg.Skia/blob/a362239968ea7201d2e376c8c16471069c1bfe54/src/Svg.Skia/Drawables/Drawable.cs#L370

Step 2 - draw anything that is affected by filter

https://github.com/wieslawsoltes/Svg.Skia/blob/a362239968ea7201d2e376c8c16471069c1bfe54/src/Svg.Skia/Drawables/Drawable.cs#L373

Step 3 - blend back the results

https://github.com/wieslawsoltes/Svg.Skia/blob/a362239968ea7201d2e376c8c16471069c1bfe54/src/Svg.Skia/Drawables/Drawable.cs#L377

A lot more goes into filters (as you can see in the SVG spec). My next task is to make SVG filters working in correct color spaces, that's not trivial either.

[ziriax]

I never got SVG working as good as HTML canvas, see e.g. https://stackoverflow.com/questions/37120158/svg-opacity-mask-resolution-and-scaling

Yes, ideally all rendering is done in 16-bit or 32-bit float per color channel linear color space. Luckily Skia supports that now.

[wieslawsoltes]

I never got SVG working as good as HTML canvas, see e.g. https://stackoverflow.com/questions/37120158/svg-opacity-mask-resolution-and-scaling

Did not yet get to quality and resolution of filters. First I need make them work correctly.

Yes, ideally all rendering is done in 16-bit or 32-bit float per color channel linear color space. Luckily Skia supports that now.

There is SKColorType.RgbaF16 and next release will include SKColorF https://github.com/mono/SkiaSharp/pull/1116 and I have to figure out how to make this things work.