How do I use the drop in mode for the viewer, in a React Three Fiber app? I get a "TypeError: Cannot read properties of undefined (reading 'quaternion')"

[newguy123-creator]

Hi I simply copied this code frorm the drop in for three js, and stuck it into my R3F app after the react imports: `import as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d'; import as THREE from 'three';

const threeScene = new THREE.Scene(); const viewer = new GaussianSplats3D.DropInViewer({ 'gpuAcceleratedSort': true }); viewer.addSplatScenes([{ 'path': './mysplat.splat' 'splatAlphaRemovalThreshold': 5 }, { 'path': '<path to .ply, .ksplat, or .splat file>', 'rotation': [0, -0.857, -0.514495, 6.123233995736766e-17], 'scale': [1.5, 1.5, 1.5], 'position': [0, -2, -1.2] } ]); threeScene.add(viewer);`

Then I installed the "[vite-plugin-cross-origin-isolation] and setup vite with this: `import { defineConfig } from "vite";

export default defineConfig({ plugins: [ { name: "configure-response-headers", configureServer: (server) => { server.middlewares.use((_req, res, next) => { res.setHeader("Cross-Origin-Embedder-Policy", "require-corp"); res.setHeader("Cross-Origin-Opener-Policy", "same-origin"); next(); }); }, }, ], });`

However, 1stly, my firewall complains: net::ERR_BLOCKED_BY_RESPONSE.NotSameOriginAfterDefaultedToSameOriginByCoep 200 (OK)

Then I get: @mkkellogg_gaussian-….js?v=fe91033c:9098 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'quaternion') at _Viewer.updateSplatSort (@mkkellogg_gaussian-…?v=fe91033c:9098:63) at finish (@mkkellogg_gaussian-…?v=fe91033c:8770:16) at @mkkellogg_gaussian-…?v=fe91033c:8806:17

Any ideas?

[newguy123-creator]

I'd also like to add that the end goal is to see if there is any difference when viewed on the Quest 1 or 2 using this repo, compared to the one that comes with R3F Drei Splats, which is based on "antimatter15/splat". Currently I'm using the Drei splat loader to load my splat file and using the Meta Browser on Quest1 and Quest 2 to view it, however its extremely jittery and laggy to the point you literally cant look at it. Is the Quest 3 any better than the Q1 or Q2 in this regard or can I expect the same jittery laggy performance than the Q1?

For example, this demo which uses this repo is NOT usable on my Quest 1: https://webxr.cz/xr.html?scene=ST&xr=vr

[newguy123-creator]

Would setting the viewer to 2D Gaussian Splatting mode be any more performant?

[Patrick-van-Halm-360Fabriek]

I've also had this issue, I used the workaround to use the normal Viewer class and using the getSplatMesh() function and add that to the scene manually. And I also had the laggy performance, I hope this can be improved in the future maybe with a percentage shown in dense area's. I also have the same target audience as you.

[newguy123-creator]

I've also had this issue, I used the workaround to use the normal Viewer class and using the getSplatMesh() function and add that to the scene manually. And I also had the laggy performance, I hope this can be improved in the future maybe with a percentage shown in dense area's. I also have the same target audience as you.

did you make your own react component with the normal viewer class or just stuck it directly into your main app after the other imports? Could you show an example code where you put it please?

[Patrick-van-Halm-360Fabriek]

export function Viewer() {
  const three = useThree();

  const viewer = new GaussianSplats3D.Viewer({
    gpuAcceleratedSort: true,
    sharedMemoryForWorkers: false,
    renderMode: RenderMode.Always,
    ignoreDevicePixelRatio: true,
    sceneRevealMode: GaussianSplats3D.SceneRevealMode.Instant,

    camera: three.camera,
    threeScene: three.scene,
    renderer: three.renderer,
  });

  viewer.addSplatScene(item, { showLoadingUI: false, progressiveLoad: false, rotation: [0,0,0,1] })
  .then(() => {
    viewer.start();
    three.scene.add(viewer.getSplatMesh());
  });
}

[newguy123-creator]

export function Viewer() {
  const three = useThree();

  const viewer = new GaussianSplats3D.Viewer({
    gpuAcceleratedSort: true,
    sharedMemoryForWorkers: false,
    renderMode: RenderMode.Always,
    ignoreDevicePixelRatio: true,
    sceneRevealMode: GaussianSplats3D.SceneRevealMode.Instant,

    camera: three.camera,
    threeScene: three.scene,
    renderer: three.renderer,
  });

  viewer.addSplatScene(item, { showLoadingUI: false, progressiveLoad: false, rotation: [0,0,0,1] })
  .then(() => {
    viewer.start();
    three.scene.add(viewer.getSplatMesh());
  });
}

Thanks, so I would import that as: import Viewer from "./Viewer.jsx" and add it to my R3D scene with: <Viewer /> but then, where do I ad the path to my splat, I dont see how then to actually specify that....

[Patrick-van-Halm-360Fabriek]

Where it says item in viewer.addSplatScene is the path/url to the splat, ply or ksplat file.

[newguy123-creator]

Where it says item in viewer.addSplatScene is the path/url to the splat, ply or ksplat file.

I dont get it. Mine is throwing a lot of errors. Here is my entire component:

import React from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";
import * as THREE from "three";
import { useThree } from "@react-three/fiber";

function GSViewer() {
    const three = useThree();

    const viewer = new GaussianSplats3D.Viewer({
        gpuAcceleratedSort: true,
        sharedMemoryForWorkers: false,
        renderMode: RenderMode.Always,
        ignoreDevicePixelRatio: true,
        sceneRevealMode: GaussianSplats3D.SceneRevealMode.Instant,

        camera: three.camera,
        threeScene: three.scene,
        renderer: three.renderer,
    });

    viewer
        .addSplatScene("./models/model_full.splat", {
            showLoadingUI: false,
            progressiveLoad: false,
            // rotation: [0, 0, 0, 1],
            position: [7, 2, -2.7],
            scale: [5, 5, 5],
        })
        .then(() => {
            viewer.start();
            three.scene.add(viewer.getSplatMesh());
        });
}

export default GSViewer;

but then I get a whole bunch of errors: RenderMode is not defined at GSViewer (GSViewer.jsx:12:21)

If I comment out the RenderMode line, I get a different error: RangeError: Invalid typed array length: 3 at new Float32Array () at _SplatParser.parseStandardSplatToUncompressedSplatArray (@mkkellogg_gaussian-splats-3d.js?v=fbaac47e:3051:24) at @mkkellogg_gaussian-splats-3d.js?v=fbaac47e:3187:38 at @mkkellogg_gaussian-splats-3d.js?v=fbaac47e:229:15

[Patrick-van-Halm-360Fabriek]

So my bad on the rendermodes. Just remove that line since it already does Always by default. For the float thing it could be an issue with the splat itself, I had this once too. Have you tried opening the Splat in the demo: https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php

If it does work there then I don't really know why it doesn't work in the three js viewer. Maybe someone else knows this.

[newguy123-creator]

So my bad on the rendermodes. Just remove that line since it already does Always by default. For the float thing it could be an issue with the splat itself, I had this once too. Have you tried opening the Splat in the demo: https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php

If it does work there then I don't really know why it doesn't work in the three js viewer. Maybe someone else knows this.

yes the splat works in the demo and also in other implementations just cant get this implementation working in React, no matter which splat I use

[mkkellogg]

Sorry for the very late reply, but I was able to create an R3F component using the following code:

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: Array<string>, options?: Array<any> }) {

    const [scene] = useState(() => new THREE.Scene())

    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: false,
        });

        const addParams: Array<any> = sources.map((source: string, index: number) => {
            const params: any = {
                path: source
            };
            Object.assign(params, (options && options.length > index) ? options[index] : {});
            return params;
        });

        scene.add(viewer);

        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        return () => {
            viewer.dispose()
                .catch((err) => {
                    console.log("Error disposing of splats viewer:", err);
                });
        };
    }, []);

    return (<primitive object={scene} />);
}

[Patrick-van-Halm-360Fabriek]

Any reason for introducing another scene object? And disposing the viewer? Could move the viewer itself to a State object @mkkellogg

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: Array<string>, options?: Array<any> }) {
    const [viewer, _] = useState(new DropInViewer({
        sharedMemoryForWorkers: false,
        showLoadingUI: false,
    }));
    useEffect(() => {
        const addParams: Array<any> = sources.map((source: string, index: number) => {
            const params: any = {
                path: source
            };
            Object.assign(params, (options && options.length > index) ? options[index] : {});
            return params;
        });

        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });
    }, []);

    return <primitive object={viewer} />; // Since its an a THREE.Group so supported by R3F
}

[mkkellogg]

@Patrick-van-Halm-360Fabriek For adding another scene object, I have no good reason at all :smile: It was just a quick example that I didn't think about too carefully, yours is definitely cleaner! This reason for disposing the viewer is that certain resources (I forget which... I think things like the web worker and GPU resources) were not getting cleaned up automatically by the garbage collector.

[Justus03]

Hi I simply copied this code frorm the drop in for three js, and stuck it into my R3F app after the react imports: `import as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d'; import as THREE from 'three';

const threeScene = new THREE.Scene(); const viewer = new GaussianSplats3D.DropInViewer({ 'gpuAcceleratedSort': true }); viewer.addSplatScenes([{ 'path': './mysplat.splat' 'splatAlphaRemovalThreshold': 5 }, { 'path': '<path to .ply, .ksplat, or .splat file>', 'rotation': [0, -0.857, -0.514495, 6.123233995736766e-17], 'scale': [1.5, 1.5, 1.5], 'position': [0, -2, -1.2] } ]); threeScene.add(viewer);`

Then I installed the "[vite-plugin-cross-origin-isolation] and setup vite with this: `import { defineConfig } from "vite";

export default defineConfig({ plugins: [ { name: "configure-response-headers", configureServer: (server) => { server.middlewares.use((_req, res, next) => { res.setHeader("Cross-Origin-Embedder-Policy", "require-corp"); res.setHeader("Cross-Origin-Opener-Policy", "same-origin"); next(); }); }, }, ], });`

However, 1stly, my firewall complains: net::ERR_BLOCKED_BY_RESPONSE.NotSameOriginAfterDefaultedToSameOriginByCoep 200 (OK)

Then I get: @mkkellogg_gaussian-….js?v=fe91033c:9098 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'quaternion') at _Viewer.updateSplatSort (@mkkellogg_gaussian-…?v=fe91033c:9098:63) at finish (@mkkellogg_gaussian-…?v=fe91033c:8770:16) at @mkkellogg_gaussian-…?v=fe91033c:8806:17

Any ideas?

calling threeScene.add(viewer) after viewer.addSplatScene creates the issue

[mkkellogg]

Just checking in, is anyone still having any specific issues? It looks like several have been brought up in this thread, and I'm not sure which (if any) have been resolved.

[Patrick-van-Halm-360Fabriek]

The original error: TypeError: Cannot read properties of undefined (reading 'quaternion') does seem resolved on my end when I revert my changes to what I had before. If I am correct this was caused by the splat already updating immediately after load, when it was possibly not in the canvas element yet. @mkkellogg

[newguy123-creator]

I will be trying the code again this week. So to get this going with react in a R3F app, the way to go is using Patrick's code from here? https://github.com/mkkellogg/GaussianSplats3D/issues/274#issuecomment-2268273025

[Patrick-van-Halm-360Fabriek]

@newguy123-creator I had some small issues in that code, since I wanted to clarify things. The following code should work, make sure to put it in the Canvas element ofcourse.

'use client'

import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any }) {
    const [viewer, _] = useState(new GaussianSplats3D.DropInViewer({
        sharedMemoryForWorkers: false,
        showLoadingUI: false,
    }));
    useEffect(() => {
        const addParams: {path: string}[] = sources.map((source: string) => ({path: source}));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });
    }, []);

    return <primitive object={viewer} />; // Since its an a THREE.Group so supported by R3F
}

Remember this doesn't Dispose the viewer if you want to also dispose the viewer on reload I would change it to the following:

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any }) {
    const [viewer, setViewer] = useState<THREE.Group>(new THREE.Group());
    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: false,
        });
        const addParams: {path: string}[] = sources.map((source: string) => ({path: source}));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, []);

    return <primitive object={viewer} />
}

[newguy123-creator]

@newguy123-creator I had some small issues in that code, since I wanted to clarify things. The following code should work, make sure to put it in the Canvas element ofcourse.

'use client'

import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any }) {
    const [viewer, _] = useState(new GaussianSplats3D.DropInViewer({
        sharedMemoryForWorkers: false,
        showLoadingUI: false,
    }));
    useEffect(() => {
        const addParams: {path: string}[] = sources.map((source: string) => ({path: source}));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });
    }, []);

    return <primitive object={viewer} />; // Since its an a THREE.Group so supported by R3F
}

Remember this doesn't Dispose the viewer if you want to also dispose the viewer on reload I would change it to the following:

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any }) {
    const [viewer, setViewer] = useState<THREE.Group>(new THREE.Group());
    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: false,
        });
        const addParams: {path: string}[] = sources.map((source: string) => ({path: source}));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, []);

    return <primitive object={viewer} />
}

Thanks Patrick. So from that, the only thing I change is ({path: source}) and I change it to something like: ({path: "./mysplat.ply"}) and then stick it into my canvas with <SplatsView /> thats it?

Also, what does the 1st line do? 'use client'

[Patrick-van-Halm-360Fabriek]

@newguy123-creator

Also, what does the 1st line do? 'use client'

That is a line specifically for server components if I recall correctly. It clarifies that the code should be ran on the client rather than the server.

Thanks Patrick. So from that, the only thing I change is ({path: source})

Example implementation

<Canvas gl={{ antialias: false }}>
    <SplatsView sources={["./mysplat.ply"]} />
</Canvas>

[newguy123-creator]

thanks @Patrick-van-Halm-360Fabriek

[newguy123-creator]

I'm trying to add position, scale, rotation to the SplatsView like so @Patrick-van-Halm-360Fabriek , but it doesnt seem to be working. How can I pass these values to the SplatsView component?

<SplatsView
    sources={["./mymodel.splat"]}
    position={[7, 2, -5.7]}
    scale={[5, 5, 5]}
    rotation-x={-Math.PI}
    rotation-y={-Math.PI}
/>

[mkkellogg]

Sorry for the delayed response, the react component would need to be updated to use the options parameter, something like:

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any[] }) {
    const [viewer, setViewer] = useState<THREE.Group>(new THREE.Group());
    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: false,
        });
        const addParams: {path: string}[] = sources.map((source: string, index: number) => (
            {
                path: source, 
                position: options[index]?.position,
                scale: options[index]?.scale,
                rotation: options[index]?.rotation
             }
        ));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, []);

    return <primitive object={viewer} />
}

And then you'd need to use the component like this:

<SplatsView
    sources={["./mymodel.splat"]}
    options={[{position: [7, 2, -5.7], scale: [5, 5, 5], rotation=[0, 0, 0, 1]}]}
/>

[newguy123-creator]

Sorry for the delayed response, the react component would need to be updated to use the options parameter, something like:

'use client'

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }: { sources: string[], options?: any[] }) {
    const [viewer, setViewer] = useState<THREE.Group>(new THREE.Group());
    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: false,
        });
        const addParams: {path: string}[] = sources.map((source: string, index: number) => (
            {
                path: source, 
                position: options[index]?.position,
                scale: options[index]?.scale,
                rotation: options[index]?.rotation
             }
        ));
        viewer.addSplatScenes(addParams, false)
            // Handle any scene loading exceptions, including early download aborts. This is important in
            // next.js apps where components get rendered twice, and the viewer gets disposed immediately
            // after the first load.
            .catch((err) => {
                console.log("Error loading splat scenes:", err);
            });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, []);

    return <primitive object={viewer} />
}

And then you'd need to use the component like this:

<SplatsView
    sources={["./mymodel.splat"]}
    options={[{position: [7, 2, -5.7], scale: [5, 5, 5], rotation=[0, 0, 0, 1]}]}
/>

How would this look in React JSX instead of TS?

[mkkellogg]

Mainly the type annotations would go away. For example: sources: string[] becomes sources and useState<THREE.Group> goes back to useState Also you'd need to remove the question marks from places like options[index]?.

[newguy123-creator]

Mainly the type annotations would go away. For example: sources: string[] becomes sources and useState<THREE.Group> goes back to useState Also you'd need to remove the question marks from places like options[index]?.

can I specify SH degree in this dropinviewer, if using the memory-optimizations branch? ...and can I just add that code, directly after the line that says "showLoadingUI: false,"?

[mkkellogg]

Yeah you can use most of the standard Viewer parameters for DropInViewer.

[mkkellogg]

Closing for now, please let me know if there are additional questions or issues.