Weird rendering behaviour using the DropInViewer

[Patrick-van-Halm-360Fabriek]

Since the issue of #274 seems resolved now I wanted to try and use the DropInViewer, however I struck a weird rendering issue.

In my case I use R3F and have a wrapper class for managing loading Splats. However once I use the DropInViewer directly I seem to have weird rendering issues.

However using my alternative code where I use a Viewer and when the mesh is loaded use the getSplatMesh() function it does render correctly, in order to achieve this I the snippet is posted below.

My question is if there is a way to render with the DropInViewer rather than using a "Hacky" way to implement it like I have.

Also don't mind the upside down rendering, this is simply fixed by changing the rotation when loading the splat scene which isn't included in the snippet.

DropInViewer

Viewer

Supersplat

My load snippet for DropInViewer

    public async load() {
        this.finishedLoading = false;
        const loading: GaussianSplats3D.AbortablePromise<any>[] = [];
        while (!this.finishedLoading) {
            if (loading.length === this.MAX_CONCURRENT_DOWNLOADS) {
                await new Promise<void>(resolve => {
                    setTimeout(resolve, 100)
                });
                continue;
            }

            const item = this.queue.dequeue();
            if (!item) {
                await new Promise<void>(resolve => {
                    setTimeout(resolve, 100)
                });
                continue;
            }

            const viewer = new GaussianSplats3D.DropInViewer({
                sharedMemoryForWorkers: false
            });
            const promise = viewer.addSplatScene(item, { showLoadingUI: false, progressiveLoad: true, })
            .then(() =>
            {
                loading.splice(loading.indexOf(promise), 1);
                this.loaded.push(item);
                this.emit("loaded", viewer, [], [], item);
                if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
            })
            .catch((e: Error) => {
                loading.splice(loading.indexOf(promise), 1);
                console.error(`Error loading ${item}, ${e}`);
                if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
            })
            loading.push(promise);
            console.log(`Loading ${item}`);
        }
    }

Load snippet for Viewer

public async load() {
    this.finishedLoading = false;
    const loading: GaussianSplats3D.AbortablePromise<any>[] = [];
    while (!this.finishedLoading) {
        if (loading.length === this.MAX_CONCURRENT_DOWNLOADS) {
            await new Promise<void>(resolve => {
                setTimeout(resolve, 100)
            });
            continue;
        }

        const item = this.queue.dequeue();
        if (!item) {
            await new Promise<void>(resolve => {
                setTimeout(resolve, 100)
            });
            continue;
        }

        const viewer = new GaussianSplats3D.Viewer({
            sharedMemoryForWorkers: false,
            selfDrivenMode: true,
            useBuiltInControls: false,
            antialiased: false,
            renderMode: RenderMode.Always,
            ignoreDevicePixelRatio: false,

            dynamicScene: true,
            sceneRevealMode: GaussianSplats3D.SceneRevealMode.Instant,
            halfPrecisionCovariancesOnGPU: true,
            integerBasedSort: true,

            camera: this.threeData?.camera,
            threeScene: this.threeData?.scene,
            renderer: this.threeData?.renderer
        });
        const promise = viewer.addSplatScene(item, { showLoadingUI: false, progressiveLoad: true, })
        .then(() =>
        {
            loading.splice(loading.indexOf(promise), 1);
            this.loaded.push(item);
            viewer.start();
            this.emit("loaded", viewer.getSplatMesh(), [], [], item);
            if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
        })
        .catch((e: Error) => {
            loading.splice(loading.indexOf(promise), 1);
            console.error(`Error loading ${item}, ${e}`);
            if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
        })
        loading.push(promise);
        console.log(`Loading ${item}`);
    }
}

[mkkellogg]

There's a lot going on here, so it's hard to say what might be causing the issue just from looking at the code :) My first suggestion is to use the same parameters when instantiating DropInViewer as you are for Viewer (except those that are irrelevant to DropInViewer such as selfDrivenMode etc...).

I would also recommend using Viewer.addSplatScenes() to load multiple splat scenes instead of multiple calls to Viewer.addSplatScene(). And for either of those functions, you need to make sure the promise returned completes before calling the function again (and I don't think that is happening in the above code?)

This is just off the top of my head, I would probably have to troubleshoot it myself to know exactly what is happening. How many scenes are you trying to load?

[Patrick-van-Halm-360Fabriek]

Currently I am only loading one scene. With the same parameters, (leaving the unnecessary ones in, since they get disregarded in the constructor of the DropInViewer) the result is absolutely the same.

public async load() {
    this.finishedLoading = false;
    const loading: GaussianSplats3D.AbortablePromise<any>[] = [];
    while (!this.finishedLoading) {
        if (loading.length === this.MAX_CONCURRENT_DOWNLOADS) {
            await new Promise<void>(resolve => {
                setTimeout(resolve, 100)
            });
            continue;
        }

        const item = this.queue.dequeue();
        if (!item) {
            await new Promise<void>(resolve => {
                setTimeout(resolve, 100)
            });
            continue;
        }

        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            selfDrivenMode: true,
            useBuiltInControls: false,
            antialiased: false,
            renderMode: RenderMode.Always,
            ignoreDevicePixelRatio: false,

            dynamicScene: true,
            sceneRevealMode: GaussianSplats3D.SceneRevealMode.Instant,
            halfPrecisionCovariancesOnGPU: true,
            integerBasedSort: true,

            camera: this.threeData?.camera,
            threeScene: this.threeData?.scene,
            renderer: this.threeData?.renderer
        });
        const promise = viewer.addSplatScene(item, { showLoadingUI: false, progressiveLoad: true, })
        .then(() =>
        {
            loading.splice(loading.indexOf(promise), 1);
            this.loaded.push(item);
            // viewer.start();
            this.emit("loaded", viewer, [], [], item);
            if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
        })
        .catch((e: Error) => {
            loading.splice(loading.indexOf(promise), 1);
            console.error(`Error loading ${item}, ${e}`);
            if (loading.length === 0 && this.queue.length === 0) this.finishedLoading = true;
        })
        loading.push(promise);
        console.log(`Loading ${item}`);
    }
}

Same happens with the following snippet inside of the Canvas element.

'use client'

import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any }) {
    const [viewer, setViewer] = useState<GaussianSplats3D.DropInViewer>(new GaussianSplats3D.DropInViewer({
        sharedMemoryForWorkers: false,
        showLoadingUI: false
    }));
    useEffect(() => {
        const addParams: {path: string}[] = sources.map((source: string) => ({path: source}));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        setViewer(viewer);
    }, []);

    return <primitive object={viewer} />
}

The PLY I am trying to load is added for debugging. PLY file expires in 7 days

The PLY was exported from Postshot, using the Splat ADC profile.

[mkkellogg]

Do you see any errors in the developer console? For what it's worth, I tested the last snippet you pasted above and it worked fine.

[Patrick-van-Halm-360Fabriek]

Negative. Did you also try with my provided PLY? Could it be something with the file?

[mkkellogg]

I did try with your PLY file and it work correctly from what I could tell. Out of curiosity, what version of my viewer are you using? I'm currently testing with the latest version (0.4.4).

[Patrick-van-Halm-360Fabriek]

Same version here, interesting how it behaves differently. I am going to try with a new project to see if it works, if not I will make it public and share it here so we can debug it.

[Patrick-van-Halm-360Fabriek]

Hmm on a reworked project it does seem to work.

[Patrick-van-Halm-360Fabriek]

I have found the issue, it occurs when there is a shadow plane used. In my app I calculate the shadow plane off the bounding box of the mesh, since the bounding box seems to not encapsulate the splat correctly it placed the shadow plane in the center causing this weird rendering behaviour.

[mkkellogg]

Ah, interesting. If you're trying to calculate the bounding box via the "normal" three.js way (Box3.setFromObject() or BufferGeometry.computeBoundingBox()) it definitely won't work correctly :) Maybe I need to implement a function for that.

[Patrick-van-Halm-360Fabriek]

Would be cool, since we use it also to also calculate the bounds for zooming into objects.

[mkkellogg]

In the memory-optimizations for which I have this PR I added the SplatMesh.computeBoundingBox() function. Would you be able to see if that works for you?

[Patrick-van-Halm-360Fabriek]

@mkkellogg Thanks for the functionality, based on my PLY file there are some inconsistencies.

The blue area is indeed detected and blocked by my camera movement. However in the red area I can completely go through and am not blocked.

Edit: no worries I am wrong, had a bad call somewhere.

[Patrick-van-Halm-360Fabriek]

Is it possible to make it so using this with .expandByObject() and .setFromObject() works by passing the DropInViewer for example, currently it does not.

[mkkellogg]

I think the way those functions work prevents them from being used with anything other than meshes with geometry that has a standard position attribute, and the splat mesh definitely does not work that way. You can see here for example that expandByObject() requires that data. You would have to write a custom Box3 to do that, maybe by inheriting from Box3 and implementing an expandBySplatMesh() and setFromSplatMesh() function.

[Patrick-van-Halm-360Fabriek]

That works, I currently just create a handler that checks upon each DropInViewer and Union the Box3. Thanks for the implementation it does make it work better!

[mkkellogg]

Glad I could help!