MaskPass breaks background update for transmissive objects

[deenns]

Currently there are the following lines of code in the MaskPass class

state.buffers.color.setMask( false );
state.buffers.depth.setMask( false );

Both values are set to false, but these operations aren't reverted at the end of the pass.

As the result, if after the scene is rendered you remove the scene background, it's not removed for transmissive objects. What's more, if there are non-transmissive parts in a transmissive model and it's being rotated, then the non-transmissive parts leave a trail on the background.

1. A box mask is applied to a scene with some background and a transmissive object. Note that there is a small non-transmissive box in the model.

2. The scene background is removed, but it's still there for the transmissive object.

3. When the object is rotated, the small opaque black box leaves a trail on the background.

Some code

function createBox(scale = 1, x = 0, y = 0, z = 0) {
    const geometry = new BoxGeometry(scale , scale , scale);
    const mesh = new Mesh(geometry);
    mesh.material = new MeshBasicMaterial({ color: 0x000000 });
    mesh.position.x = x;
    mesh.position.y = y;
    mesh.position.z = z;
    return mesh;
}

const maskScene = new Scene();
maskScene.add(createBox(3));

// load a model and add an opaque box to it
gltf.scene.add(createBox(0.1, -0.5, 0.5, 0));
scene.add(gltf.scene);

const maskRenderTarget = new WebGLRenderTarget(500, 500, {
    stencilBuffer: true,
    samples: 8,
});
const composer = new EffectComposer(renderer, maskRenderTarget);
composer.setPixelRatio(window.devicePixelRatio);

// passes
composer.addPass(new ClearPass());
composer.addPass(new MaskPass(maskScene, camera));

const renderPass = new RenderPass(scene, camera);
renderPass.clear = false;
composer.addPass(renderPass);

composer.addPass(new ClearMaskPass());
composer.addPass(new ShaderPass(GammaCorrectionShader));

Probable solution

If the following code

state.buffers.color.setMask( true );
state.buffers.depth.setMask( true );

is executed before RenderPass (e.g. at the end of MaskPass), then the problem disappears. For this very issue only the first line is enough.

[Mugen87]

I'm confused why the additional setMask() calls are required. They were originally not added in MaskPass since it should be sufficient to just unlock color and depth buffer. WebGLState.setMaterial() is called per render item and executes setMask() for both color and depth buffer:

https://github.com/mrdoob/three.js/blob/7fa8637df3edcf21a516e1ebbb9b327136457baa/src/renderers/webgl/WebGLState.js#L757-L758

Meaning as long as the buffers are unlocked and depthWrite and colorWrite are set to true, the WebGL commands gl.colorMask() and gl.depthMask() will be called with true as well.

[Mugen87]

Wait, I think I understand now. Color and depth buffer can only be cleared if they are writable. setMask() is potentially called to late which breaks the clear.