[filter-effects-1] Scale image before or after filter() is applied?

[faceless2]

Opened to continue the discussion from https://github.com/w3c/csswg-drafts/issues/4706

There are two ways to conceive of the filter() function when combined with scaling:

1. The filter is applied to the image, then the image is scaled to the desired size.
2. The image is scaled to the desired size, then the filter is applied. This requires thinking of filter(nnn) as generating an image with filtering instructions - instructions which are applied on composition.

These are equally valid viewpoints, as evidenced by the fact that Safari currently uses both approaches!

• background-image and mask-image are scaled to the size set by background-size, then filtered.
• border-image is filtered first, then scaled to its final size.

(example demonstrating this at https://jsbin.com/xisohum/edit?css,output)

I think it's necessary to settle on one of these approaches, and I have come to the strong conclusion that option 2 is best:

1. It's consistent with how the combination of filter and image-set() or image-resolution already works.
2. It means the units specified by the user in the filter (e.g. blur(5px)) are used exactly as specified.
3. It makes it easier to get an identical filter applied to multiple background-images of different sizes, or background and border images (which are scaled using different approaches).
4. Scaling as part of border-image is non-trivial. Achieving a specific result from a filter if option 1 was chosen would demand quite a bit of work from the author.

In other words, I'd expect no difference in visual appearance from any of these operations:

<img src="1000x1000.png" style="image-resolution: 5dppx; filter: blur(2px)"/>
<div style="content: image-set(1000x1000.png 5dppx); filter: blur(2px)"/>
<div style="content: filter(image-set(1000x1000.png 5dppx), blur(2px))"/>
<div style="background: url(1000x1000.png) top left / 200px 200px; filter: blur(2px)"/>
<div style="background: filter(url(1000x1000.png), blur(2px)) top left / 200px 200px"/>

I've created a PR for some tests that assume option 2 at https://github.com/web-platform-tests/wpt/pull/23429/files, but there's probably some discussion to be had on whether this is actually the approach that we want. And whatever the decision, it needs documenting.

[AmeliaBR]

My first instinct was definitely concern that this approach muddies the “a filter function returns an image” data model, but on reflection I think it works. We already have things like SVG images and gradients which can't actually be rendered until we know what size they are going to be drawn at. So this would be similar: the filter function returns a vector image consisting of an embedded image file, plus a size-dependent filter operation.

I do agree that this makes things much simpler for authors, especially for cases like border image where you may have different scaling factors for different parts of the image, or along different axes.

It's also worth considering whether the “image with filtering instructions” approach could be extended to interact with image tiling (so the image is tiled and then blurred — or at least blurred after the browser knows how it is going to be tiled & can make the correct adjustments). This would simplify the currently spec'd version which requires a blur in a filter function to behave differently than the blur elsewhere. (Note, however, that this part definitely would muddy up the separation between the image and the context in which it is used.)

[tabatkins]

I'm pretty sure I've come to strongly agree - 2 is the better solution. It's more likely to match what the author wants/expects, more of the time. It also means that if we still do a background-filter that applies to the whole composited stack, units in filters will be interpreted identically between the two cases, which I like.

It's also worth considering whether the “image with filtering instructions” approach could be extended to interact with image tiling (so the image is tiled and then blurred — or at least blurred after the browser knows how it is going to be tiled & can make the correct adjustments).

This was discussed, and it's easy to achieve either way, and can be made automatic with only a tiny bit of context-dependent smarts - if an image is tiled, you use the "wraparound" bleed mode.

But that's honestly not different from just doing the whole filter on the post-tiled image, alone with all the other filters.

(Plus, I think applying the filter to post-tiled images actually is theoretically distinguishable? If a blur would visibly spread more than one tile away, I think using wraparound on a single tile then tiling, vs tiling then blurring the whole thing, is distinguishable.)

[AmeliaBR]

if an image is tiled, you use the "wraparound" bleed mode.

Note that the current spec is to use the “duplicate" edge mode for blurs inside filter(), which provides acceptable results for both tiled and untiled images (you don't get semi-transparent edges on an opaque image, but you also don't get blurring across the edges).

(Sorry for confusing matters by not digging up that spec before commenting.)

Adding a context-based rule to switch to "wrap" mode for tiled images would probably give better results for fully tiled textures, but that doesn't address background images that repeat in one direction only, or the more complicated tiling corner effects in border images. So I think that if we're going to break the encapsulation in that way, might as well make it smart enough to blur after tiling / composing the full border image. Browsers could still optimize and blur before tiling if the calculate that it's “safe” based on the blur radius.