TravisThomp commented 5 months ago

I've been having this issue when applying a rotation to an object, It works fine on smaller scenes, but once the scene consists of a large amount of gaussians. The scene begins to distort, inverse rotation at times and clip on itself, and consumes all of the ram available (tldr its a shit show lol).

Example Video

https://github.com/huggingface/gsplat.js/assets/50000930/8b2b89bd-c758-4f27-be0c-4fb9428c44e7

Ram Usage Video

https://github.com/huggingface/gsplat.js/assets/50000930/cf96640c-79ec-4211-b495-d1b1ced06917

This only occurs on firefox browsers (Tested on Version 122.0)

Works fine on Chrome
Works fine on Edge

Notes

Tested on two devices, for redundancy -> firefox issue only
Scene fps is far lower on firefox(with bug)
DOES NOT OCCUR WITH STATIONARY SCENES
Taking snapshots of memory usage on firefox does not show any additional memory being allocated (only +-50mb for this scene)
Speed of rotation does not effect speed of ram being consumed

Current Theory

There is an overflow that occurs in the translation of the gaussians since this only occurs when the scene is large. I'm assuming that the other browsers(chrome, edge) are cleaning this up automatically causing it to work fine with them, but firefox doesn't do this?

Code to recreate (pretty much any scene with a lot of gaussians rotating)

<script lang="ts">
    import { onMount } from "svelte";
    import * as SPLAT from "gsplat";
    import { Splat, Camera, Scene } from "gsplat";

    function loadModel(scene: Scene): Promise<Splat> {
        let modelURL =
            "https://huggingface.co/datasets/dylanebert/3dgs/resolve/main/bonsai/bonsai-7k.splat";

        let model = SPLAT.Loader.LoadAsync(modelURL, scene, () => {});

        return model;
    }

    function updateModelRotation(model: Splat, tick: number) {
        const autoRotateMultipler = 0.025;

        // ERROR CAUSED BY THIS LINE(atleast it doesnt show when commented out)
        model.rotation = SPLAT.Quaternion.FromEuler(
            new SPLAT.Vector3(0, tick * autoRotateMultipler, 0),
        );
    }

    onMount(async () => {
        const canvas = document.getElementById("canvas") as HTMLCanvasElement;

        const renderer = new SPLAT.WebGLRenderer(canvas, null);
        canvas.style.background = "black";

        const scene = new SPLAT.Scene();
        const camera = new SPLAT.Camera();
        camera.rotation = SPLAT.Quaternion.FromEuler(
            new SPLAT.Vector3(0, 0, 0),
        );

        let model = await loadModel(scene);

        model.scale = new SPLAT.Vector3(2, 2, 2);

        let tick: number = 0;
        const frame = () => {
            updateModelRotation(model, tick);

            tick += 1;
            renderer.render(scene, camera);
            requestAnimationFrame(frame);
        };

        const handleResize = () => {
            renderer.setSize(canvas.clientWidth, canvas.clientHeight);
        };
        requestAnimationFrame(frame);

        window.addEventListener("resize", handleResize);
    });
</script>

<canvas id="canvas"> </canvas>

<style>
    canvas {
        width: 100vw;
        height: 100vh;
    }
</style>

mateusz-malicki commented 5 months ago

@TravisThomp after setting model.rotation you should invoke model.applyRotation(). Also, there is no point in increasing your tick - just set how much you want to rotate (but this value should be scaled to time from last render).

        var rotation = SPLAT.Quaternion.FromEuler(new SPLAT.Vector3(0, 0.1, 0));
    const frame = () => {
        model.rotation = rotation;
        model.applyRotation();
        renderer.render(scene, camera);
        requestAnimationFrame(frame);
    };

btw, I think that moving camera instead of rotating splat is better idea.