Compatibility between Viewer & PostProcessing Effects

[tansh-kwa]

Does the viewer currently support postprocessing effects in threejs. I've added elements to the scene that I would like to post-process with the Composer object. This contains a reference to the renderer, however, passing the Composer directly to the dropInViewer causes issues the errors. Calling .render() on the composer in the update function causes the splat to disappear.

I suppose I want to ask if there is anything to look out for w.r.t compatibility with threejs post processing.

[tansh-kwa]

I believe this could have something to do with PostProcessing effects that are using the DepthBuffer. I am seeing that a shaderMaterial that uses the depthBuffer is setting everything that is not a mesh object to black. To add more context to this: I have a .ply loaded into the viewer and a mesh object that I am rendering in the scene. I am trying to use ThreeJS's outlinePass to render the mesh's outlines, and cleaning the antialiased edges with effectFXAA. Adding effectFXAA removes the splat. Not having FXAA means the outline appears but is slowly disappearing as the splat loads. I've attached videos to show what is happening:

The outline fading: https://github.com/mkkellogg/GaussianSplats3D/assets/16449141/b80ffc5f-c80d-4e8f-859a-6134a43487ca

The outline visible against black backgroun/through a 'hole' in the splat:

https://github.com/mkkellogg/GaussianSplats3D/assets/16449141/b2807b68-81c0-49c2-9174-7753c1f94962

[mkkellogg]

You definitely have to be careful with post-processing effects, especially when they depend on depth buffer data; they're not a one-size-fits-all solution. The issue with combining splats and post-processing results is that the splats are always rendered as transparent objects (no depth-buffer data) and if the results of the post-processing are also transparent, then there will be no way to guarantee proper rendering order (because of the lack of depth buffer data). Often there can be workarounds, but they will be very specific to the effect you are trying to achieve. Would you be able to share your code? Maybe I can figure out a quick solution.

[tansh-kwa]

Sure thing, I've attached a trimmed down version of what I am running to showcase the issue. The files I'm using are linked here too: fused.ply: http://bashupload.com/2Grv7/fused.ply cameras.json: https://pastebin.com/9hE4SAYX transforms.json: https://pastebin.com/M9dJwUgQ

import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
import * as THREE from 'three';
import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
import { TransformControls } from 'three/addons/controls/TransformControls.js';
import CameraControls from 'camera-controls';
import _, {map} from 'underscore';

// For shader effects, plz don't explode
import { EffectComposer } from "three/examples/jsm/postprocessing/EffectComposer.js";
import { RenderPass } from "three/examples/jsm/postprocessing/RenderPass.js";
import { ShaderPass } from "three/examples/jsm/postprocessing/ShaderPass.js";
import { FXAAShader } from "three/examples/jsm/shaders/FXAAShader.js";
import { OutlinePass } from "three/examples/jsm/postprocessing/OutlinePass.js";

CameraControls.install( { THREE: THREE } );
const clock = new THREE.Clock();

const renderWidth = window.innerWidth;
const renderHeight = window.innerHeight;

const rootElement = document.createElement('div');
rootElement.style.width = renderWidth + 'px';
rootElement.style.height = renderHeight + 'px';
document.body.appendChild(rootElement);

const renderer = new THREE.WebGLRenderer({
    antialias: false,
    alpha: true,
    preserveDrawingBuffer: true // this is the thing that causes the streaks nightmare, ugh. 
});

const scene = new THREE.Scene();

renderer.setSize(renderWidth, renderHeight);
rootElement.appendChild(renderer.domElement);

const camera = new THREE.PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500);
console.log("camera instance: ", camera);
camera.position.copy(new THREE.Vector3().fromArray([0, 0, 6]));
camera.up = new THREE.Vector3().fromArray([0, -1, 0]).normalize();
// camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));

const cameraControls = new CameraControls( camera, renderer.domElement );
console.log(cameraControls);
// rendering passes for effects
const depthTexture = new THREE.DepthTexture();

let selectedObjects = [];
const outlinePass = new OutlinePass(new THREE.Vector2(window.innerWidth, window.innerHeight), scene, camera, selectedObjects);
outlinePass.renderToScreen = true;
outlinePass.selectedObjects = selectedObjects;

let params = {
    edgeStrength: 5,
    edgeGlow: 1,
    edgeThickness: 1.0,
    pulsePeriod: 0,
    usePatternTexture: true
};

outlinePass.edgeStrength = params.edgeStrength;
outlinePass.edgeGlow = params.edgeGlow;
outlinePass.visibleEdgeColor.set(0xff0000);
outlinePass.hiddenEdgeColor.set(0xff0000);

const effectFXAA = new ShaderPass(FXAAShader);
effectFXAA.uniforms["resolution"].value.set(
    1 / window.innerWidth,
    1 / window.innerHeight
  );

  const viewer = new GaussianSplats3D.Viewer({
    'selfDrivenMode': false,
    'renderer': renderer,
    'camera': camera,
    'threeScene': scene,
    'useBuiltInControls': false,
    'ignoreDevicePixelRatio': false,
    'gpuAcceleratedSort': true,
    'halfPrecisionCovariancesOnGPU': true,
    'sharedMemoryForWorkers': true,
    'integerBasedSort': true,
    'dynamicScene': true,
    'webXRMode': GaussianSplats3D.WebXRMode.None,
    'renderMode': GaussianSplats3D.RenderMode.OnChange
});

const renderTarget = new THREE.WebGLRenderTarget(
    window.innerWidth,
    window.innerHeight,
    {
      depthTexture: depthTexture,
      depthBuffer: true,
      minFilter: THREE.LinearFilter, 
      magFilter: THREE.LinearFilter, 
      format: THREE.RGBAFormat, 
      stencilBuffer: false
    }
  );

const composer = new EffectComposer(viewer.renderer, renderTarget);
const pass = new RenderPass(scene, camera);
composer.addPass(pass);
composer.addPass(outlinePass);
// composer.addPass(effectFXAA);

let scale_factor;
let transform_data;

let pointer = new THREE.Vector2();
let frame;
let flag = false;

// Transform controls for controlling position and rotation of GLTF file - used for alignment
const transformControls = new TransformControls(camera, renderer.domElement)
transformControls.size = 0.5;
scene.add(transformControls)

// disable view controls when transform controls are enabled.
transformControls.addEventListener('mouseDown', function () {
    cameraControls.enabled = false
})
transformControls.addEventListener('mouseUp', function () {
    cameraControls.enabled = true
})

async function load_data(data_path){
    const response = await fetch(data_path);
    let raw = await response.json();
    return raw;
}

function sphere_from_position(position, _scale_factor, _color){
    const geometry = new THREE.SphereGeometry(0.005, 32, 16);
    const material = new THREE.MeshBasicMaterial({
    color: _color
    });
    const sphere = new THREE.Mesh(geometry, material);
    // sphere.position.set(position[0],position[1],position[2]);
    let adjusted_x = position[0] * _scale_factor;
    let adjusted_y = position[1] * _scale_factor;
    let adjusted_z = position[2] * _scale_factor;
    // console.log([adjusted_x, adjusted_y, adjusted_z])
    sphere.position.set(adjusted_x, adjusted_y, adjusted_z);
    scene.add(sphere);
    selectedObjects.push(sphere);
}

function init_camera_position(position){
    let center = new THREE.Vector3();
    viewer.splatMesh.getSplatCenter(1, center, false);

    cameraControls.setPosition( 0.21536377373778526,  0.27769848917492385,  0.8083895558789334)
    // camera.position.set(0, -1.1102230246251565e-16,1.40078068416906)
    // console.log(position);
    // camera.position.set(position[0], position[1], position[2]);
    // cameraControls.setLookAt(position[0]*scale_factor,
    //     position[1]*scale_factor,
    //     position[2]*scale_factor,
    //     -1.32627478*scale_factor, 
    //     -0.55167136*scale_factor,  
    //     3.4691239*scale_factor,
    //     true); //lookat is from corners
}

fetch('splats/med-res/transforms.json')
    .then((response) => response.json())
    .then((json) => {
        console.log(json);
        transform_data = json;
        scale_factor = json['scale_factor'];
        viewer.addSplatScene('splats/med-res/fused.ply', {
            'scale':[scale_factor, scale_factor , scale_factor]
        })
        .then(() => {
            // selectedObjects.push(viewer.splatMesh);
            requestAnimationFrame(update);
            start_splat_actions();
        });
    })

function start_splat_actions(){
    let center = new THREE.Vector3();
    viewer.splatMesh.getSplatCenter(1, center, false);
    console.log("viewer instance: ", viewer);
    cameraControls.setOrbitPoint(center.x, center.y, center.z);

    let splat_cameras = load_data("splats/med-res/cameras.json");
    splat_cameras.then((cameras)=>{
        for(let c of cameras){
            sphere_from_position(c.position, scale_factor, 0xffff00);
        }
        let camera_init = _.sample(cameras, 1)[0];
        init_camera_position(camera_init.position);
    })
    const axesHelper = new THREE.AxesHelper( 1 );
}

function raycast(){
    let renderDimensions = new THREE.Vector2();
    let intersectPoint = new THREE.Vector3();
    let outHits = [];
    getRenderDimensions(renderDimensions);
    viewer.raycaster.setFromCameraAndScreenPosition(camera, pointer, renderDimensions);
    let r = viewer.raycaster.intersectSplatMesh(viewer.splatMesh, outHits);
    if (outHits.length > 0) {
        const hit = outHits[0];
        const intersectionPoint = hit.origin;
        intersectPoint.copy(intersectionPoint);
    }
}

function getRenderDimensions(outDimensions) {
    renderer.getSize(outDimensions);
}

function onPointerMove( event ) {
    pointer.x = event.offsetX;
    pointer.y = event.offsetY;
}

window.addEventListener('click', ()=>raycast())
window.addEventListener( 'pointermove', onPointerMove );

function update() {
    renderer.clear();
    const delta = clock.getDelta();
    const hasControlsUpdated = cameraControls.update( delta );
    // cameraControls.azimuthAngle += 20 * delta * THREE.MathUtils.DEG2RAD
    requestAnimationFrame(update);
    composer.render();
    viewer.update();
    viewer.render();
}

function onWindowResize() {
    camera.aspect = window.innerWidth / window.innerHeight;
    camera.updateProjectionMatrix();

    renderer.setSize(window.innerWidth, window.innerHeight);
    composer.setSize(window.innerWidth, window.innerHeight);
    effectFXAA.setSize(window.innerWidth, window.innerHeight);
    outlinePass.setSize(window.innerWidth, window.innerHeight);
    effectFXAA.uniforms["resolution"].value.set(
      1 / window.innerWidth,
      1 / window.innerHeight
    );
  }

window.addEventListener("resize", onWindowResize, false);

[mkkellogg]

So I think I may understand what's happening. When you don't have the FXAA pass it's pretty straightforward -- the outline effect gets rendered first, but since there's no depth data associated with it (since it's transparent), the splats get rendered over it (because they get rendered second). The non-transparent scene objects (the yellow spheres) are still visible because they have depth data associated with them and the splat renderer performs a depth test during rendering.

For the case when you have FXAA enabled, my guess is that the result of the FXAA render pass is stored in renderTarget , since renderToScreen is not set for that pass. However the splat renderer doesn't know about any of that it just renders the splats to the default render buffer.

Is this all your code? How does renderTarget ultimately get displayed?

[tansh-kwa]

I see, I see, that makes sense. So if there were a way to have the outline effect write to the depth buffer they would be rendered over the splat? This is interesting since the OutlinePass also renders hidden edges which would mean rendering "through" the depthBuffer.

As for the FXAA pass, I am still trying to wrap my head around the post-processing steps and don't know what all I need to do, that is also why I have commented it out for now. This is all my code, could you explain what you mean by renderTarget being displayed? Is this something that can allow the 2D effect of outlines to be rendered "on top" of the scene

[mkkellogg]

Well I'm just guessing, but I think the output of the FXAA pass will be rendered to renderTarget and not the screen, because it is designated as the output render target for the composer:

const composer = new EffectComposer(viewer.renderer, renderTarget);

Whereas the output of the outline pass gets rendered to the screen because of this line:

outlinePass.renderToScreen = true;

So I was just wondering how you actually see the output of the FXAA pass? (Disclaimer: I am not very familiar with how the effect composer works; I have never used it myself).

[tansh-kwa]

I am able to see the output of the FXAA pass, I think the effectComposer sets the final pass to renderToScreen by default. Unfortunately I cannot run a post processing effect on the mesh objects in the scene without loosing the splat objects. It is a bit of a long shot, but I have managed to get something similar working with the LumaWebGL renderer, I'm curious if there's differences with how Threejs views splats in that renderer compared to this one.

[tansh-kwa]

Do you know if the EffectComposer "Sees" the Splat added by the viewer. I tried a simple effect composer example with only a render pass, and the splat is once again gone. Seems rendering the composer after the viewer's own render function will 'overwrite' the drawn splats.

[mkkellogg]

It definitely makes sense that when you render the composer after the viewer renders, the splat is gone. I assume the composer clears the render buffer before performing its own rendering. Could you turn off the autoClear property of the renderer before the composer renders and restore after? Maybe something like:

renderer.clear(true, true, true);
viewer.update();
viewer.render();
renderer.autoClear = false;
composer.render();
renderer.autoClear = true;

I added the renderer.clear(true, true, true); line because the viewer disables the auto clear before it renders, and restores it after.

[tansh-kwa]

Same effect ):, the only things being shown are added meshes and the post-processed outlines.

[mkkellogg]

So I've discovered a couple of things. Three.js' EffectComposer is meant to be used for all rendering, so you shouldn't be trying to call viewer.render() and then following that up with composer.render(). What really needs to be done is figure out a nice way to make rendering a splat one of the composer's passes, so you could just called composer.render() and it would render everything... that will take a bit of work :)

As for why adding the FXAA pass makes the splats go away, it looks like that pass writes to the depth buffer (which doesn't make sense to me) and then when the viewer renders the splats, none of it is visible because according to the depth buffer it should all be occluded. I found that adding the line effectFXAA.material.depthWrite = false; makes the splats visible.

However this doesn't really solve the issue of how to properly use the effect composer with the splat viewer. I don't think there is a good way to do that right now, I'm going to have to spend some time working on that.

[tansh-kwa]

Thanks very much for looking into this Mark! I know for my work it will be useful.

[tansh-kwa]

Would it be possible for you to share the your code that leaves the splat visible here?

[mkkellogg]

Just make sure your code to add passes looks like:

    const composer = new EffectComposer(renderer, renderTarget);
    const pass = new RenderPass(threeScene, camera);
    composer.addPass(pass);
    composer.addPass(outlinePass);
    composer.addPass(effectFXAA);
    effectFXAA.material.depthWrite = false;

The last line is the important one. Then make sure you're calling composer.render() before viewer.render(). As for calling composer.render() after viewer.render(), that just won't work because the composer is clearing the screen. Again this is not a real solution, a real solution will require more work :)

[tansh-kwa]

Lovely, Thanks once again Mark. In case this is helpful to anyone in the future. I managed to get the effect I wanted with an overlaid post processing effect by adding another rendering pass to the EffectComposer like so:

   const composer = new EffectComposer(renderer, renderTarget);
    const pass = new RenderPass(threeScene, camera);
    const pass = new RenderPass(viewer.splatMesh, camera);
    composer.addPass(pass);
    composer.addPass(outlinePass);

And then calling composer.render after viewer.update. Not sure if this will bite my later on, but it did the trick this time.

[mkkellogg]

Awesome, that's really good to know!

[mkkellogg]

I'm going to close this for now, but I think I might eventually add something to the README about doing stuff like this. Thanks again for digging into this!