Antialiasing support for brushes

[Shnatsel]

Right now MyPaint doesn't seem to have any kind of antialiasing in place, which makes it almost unsuitable for inking because of brush scaling issues: thicker strokes become blurry and thinner strokes become pixelated. And to make things worse, erasers suffer these issues too.

This is how brush scaling currently works in MyPaint 1.1 with Kabura brush: http://i.imgur.com/9sUQOsv.png

And this is how "Ink_gpen_25" brush in Krita 2.9 scales: http://i.imgur.com/FCTuYwD.png

Krita seems to have a separate anti-aliasing preset for brushes. Perhaps MyPaint should get something like this as well?

[0ion9]

IMO Mypaint's existing Antialiasing setting addresses this pretty well, though it's probably fair to say we could tune its value better for builtin brushes. Is there some problem with that setting for you?

[achadwick]

Does just increasing the hardness of that kabura brush to ~0.9 and dropping its pixel feather (= anti_aliasing internally) to ~0.5 help here? Possibly not in both directions, I guess...

Some old but still good documentation: https://github.com/mypaint/mypaint/wiki/Brushlib (doesn't cover anti_aliasing, I think).

[Shnatsel]

Unfortunately I cannot test that because I'm on a localized system and MyPaint's brush dialog ignores my exported LC_ALL=C that's supposed to drop it back to English for whatever reason (the rest of the program doesn't), so I cannot be sure I'm tweaking the right settings - there are so many of them, and translations are not exactly reliable.

[0ion9]

A screenshot is one way to get around that, since the ordering of the settings remains the same. (ie. if you can't identify a setting by name, if you have a current screenshot, you should still be able to identify it by index, like 'the third setting in the second section'.)

I can't provide a screenshot ATM due to bad interactions with GTK3 upgrade breaking MyPaint (mypaint issue #424), but I guess Andrew or another 'regular' here might be willing to.

[achadwick]

You can search (and correct!) localized strings for settings at https://hosted.weblate.org/projects/mypaint/ :D

[Shnatsel]

Okay, I've increased Kabura hardness to 0.9 and played with its pixel feather. Here are samples of strokes with different pixel feather values: http://i.imgur.com/7NCmbAA.png

High anti-aliasing (like 4.0) is actually pretty good, but has a weird side effect of producing a thin pixelated line inside a much thicker antialiased one from low-pressure strokes.

Hopefully I've been dialing the right knob: http://i.imgur.com/YWMLR56.png

[achadwick]

@Shnatsel So, would you say that this is a fault with the standard MyPaint Kabura brush?

I want to turn this issue into something that can be fixed with a concrete code commit, but right now I don't think that this warrant any changes to libmypaint.

[Shnatsel]

I'd say the problems are:

1) Standard brushes use hardness settings instead of pixel feather preset (anti-aliasing), because

2) Pixel feather as of mypaint 1.2 is not good enough to be usable.

Specifically:

Low pixel feather values only anti-alias thin areas of a line, leaving thick parts pixelated:

High pixel feather values make thin parts of the line disappear completely and create a thin pixelated line in a blurry halo at the transition from thin to thick line:

A good example of anti-aliasing can be seen in Krita starting with 2.9 release, specifically Inkgpen* brushes.

[achadwick]

@Shnatsel Okay, then somebody with a better understanding of this than me will have to dive into the code and figure out the libmypaint aspects of this. @jonnor and @martinxyz, you around?

It's worth knowing that the stock brushes will be split out into their own module shortly.

[martinxyz]

Still around, but not for diving ;-)

I think part of the difficulty is because brushlib by design works by blending faint dabs on top of each other. The interaction of individual dabs creates an alpha fall-off at the stroke border that is very hard to predict because it depends on too many non-constant parameters (radius, dab density, opacity, elliptical ratio).

(This is in contrast to a design with a temporary stroke layer, where individual dabs have their alpha-channels MAX'ed instead of OVER-blended. MAX'ing creates a very predictable alpha fall-off. But it also creates an unnatural change in the stroke's self-overlap whenever the layer is merged down.)

[achadwick]

Worth reading along with #9 for anyone willing to pick this up.

[briend]

master now has "Base Brush Radius" which, with some tuning, should mostly fix these issues. Basically if pixel feathering makes your brush disappear when it gets too small, you can map the base brush radius input it so that smaller brushes have less pixel feathering. If you're using dynamic brush radius w/ pressure, for instance, this input won't help. However, you can just as well map pressure to pixel feathering to achieve the same thing. I'll do a demo soon and close this

[jplloyd]

The exact behavior requested is unlikely to be implemented in libmypaint anytime soon (if at all).

To get a reasonable approximation of the desired inking brush behavior, set hardness to a constant max, and map the pixel feathering (kinda-sorta anti-aliasing) to the pressure input such that thin strokes don't get blurred away completely (map it so that low pressure results in less feathering).

In general, if you want a brush that spans a large range of radiuses/radii, it might be better to just create a few different brushes tweaked for specific radius ranges.