New blend function that replaces only completely transparent pixels

[deenns]

I want to apply certain effects to a model, but it has a shadow plane, that should not be affected. My problem is quite similar to the one described here https://github.com/pmndrs/postprocessing/issues/303

And using this example https://codesandbox.io/s/quirky-albattani-khdqq?file=/src/App.js I almost managed to solve it, but there is still an issue.

I'm using multisampling for AA and the following steps:

this.renderer = new WebGLRenderer({
    powerPreference: 'high-performance',
    antialias: false,
    stencil: false,
    depth: false,
    alpha: true, // we need transparent background
});

this.composer = new EffectComposer(this.renderer, {
    frameBufferType: HalfFloatType,
    multisampling: 4,
});

const copyPass = new CopyPass();
copyPass.texture.format = RGBAFormat;
const textureEffect = new TextureEffect({
    blendFunction: BlendFunction.ALPHA,
    texture: copyPass.texture,
});

this.composer.addPass(new ClearPass());
this.composer.addPass(new LambdaPass(() => this.camera.layers.set(0))); // render everything without the shadow plane
this.composer.addPass(new RenderPass(this.scene, this.camera));
this.composer.addPass(new EffectPass(this.camera, this.saturationEffect));
this.composer.addPass(copyPass);

this.composer.addPass(new ClearPass());
this.composer.addPass(new LambdaPass(() => this.camera.layers.set(1)));  // render the shadow plane
this.composer.addPass(new RenderPass(this.scene, this.camera));
this.composer.addPass(new EffectPass(this.camera, textureEffect)); // add the model rendered before

But a model has a whitish border where it's blended with the shadow:

I suppose it happens because after AA model edges are not completely opaque and they are mixed with the shadow. It happens even if no effects are applied. If I remove the multisampling option, then it disappears. Tried to use SMAAEffect, but for me it's worse than multisampling.

I think it can be solved with some new blend function that will replace only transparent pixels of the source image with pixels from a texture. If there is such a function, I will render the shadow into a separate texture and then blend it with the model (and not vice versa, as it is now). It can be named like DST_OPAQUE_SRC_TRANSPARENT.

Can it be done? Maybe some custom shader can be used here? Or a completely different solution?

And thank you for the library, it works really well.

[vanruesc]

I suppose it happens because after AA model edges are not completely opaque and they are mixed with the shadow.

Yea, this is caused by transparency + AA. Unfortunately, it also applies to other convolution/blur effects like DoF. Related issues: #529, #387

I don't know if this can be fixed because these artifacts are introduced as soon as AA blends non-existing pixels. The severity of the AA-haloing seems to depend on the background color of the canvas and if there's anything behind the canvas such as a background image, then there's no way to blend with that because you can't read/see it within the WebGL context.

I've updated the sandbox with a reproduction of the issue: https://codesandbox.io/s/nifty-pine-lqzhq5?file=/src/App.js

[vanruesc]

Closing this because I don't think this can be fixed in WebGL. Postprocessing and transparency don't mix well together.