mkkellogg / GaussianSplats3D

Three.js-based implementation of 3D Gaussian splatting
MIT License
1.1k stars 135 forks source link

Is it compatible with React ? #165

Closed cyrilpreiss closed 3 months ago

cyrilpreiss commented 3 months ago

Is it possible to use your library with react ?

do you have any exemple ?

cyrilpreiss commented 3 months ago

I just did a quick test and got this error:

./node_modules/@mkkellogg/gaussian-splats-3d/build/gaussian-splats-3d.module.js 5:16 Module parse failed: Unexpected token (5:16) File was processed with these loaders:

any help ?

mkkellogg commented 3 months ago

Can you post the code you're using to load and run the viewer? I can try to debug from there.

cyrilpreiss commented 3 months ago

Hi, i actually have the error as soon as i try to import:

import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';

cyrilpreiss commented 3 months ago

the problem might come from my react app, i'm trying with a new one an will let you know

cyrilpreiss commented 3 months ago

i'm almost there... but the processing splat keeps spining forver.

here is my code:

`import React, { Component } from 'react'; import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';

import { Vector3, Scene, WebGLRenderer, PerspectiveCamera, Mesh, MeshBasicMaterial, BoxGeometry } from 'three';

class GaussianSplatComp extends Component { constructor(props) { super(props);

  this.update = this.update.bind(this);

   this.state = {

  }; 

}

init(){

const {renderer, renderWidth, renderHeight} = this.setupRenderer();
this.renderer = renderer;
this.camera = this.setupCamera(renderWidth, renderHeight);
this.threeScene = this.setupThreeScene();
this.controls = this.setupControls(this.camera, this.renderer);

this.viewer = new GaussianSplats3D.DropInViewer();
this.viewer.addSplatScenes([

/ { //'path': 'assets/data/bonsai/bonsai_trimmed.ksplat', 'path': 'https://data2.gigascope.net/splats/bonsai_trimmed.ksplat', 'splatAlphaRemovalThreshold': 20, },/ { 'path': 'https://data2.gigascope.net/splats/bonsai_trimmed.ksplat', 'rotation': [-0.14724434, -0.0761755, 0.1410657, 0.976020], 'scale': [1.5, 1.5, 1.5], 'position': [-3, -2, -3.2], 'splatAlphaRemovalThreshold': 20, } ], true); this.threeScene.add(this.viewer);

requestAnimationFrame(this.update);

}

update() {

   // if (this.controls!=undefined){
    requestAnimationFrame(this.update);
    this.controls.update();
    this.renderer.render(this.threeScene, this.camera);
  //  }

}

setupRenderer() { const renderWidth = 800; const renderHeight = 600;

const rootElement = document.createElement('div');
rootElement.style.width = renderWidth + 'px';
rootElement.style.height = renderHeight + 'px';
rootElement.style.position = 'relative';

// rootElement.style.left = '50%'; rootElement.style.top = '50%'; // rootElement.style.transform = 'translate(-50%, -50%)';

document.getElementById("GSContainer").appendChild( rootElement );

// document.body.appendChild(rootElement);

const renderer = new WebGLRenderer({
    antialias: false
});
renderer.setSize(renderWidth, renderHeight);
rootElement.appendChild(renderer.domElement);

return {
  'renderer': renderer,
  'renderWidth': renderWidth,
  'renderHeight': renderHeight
}

}

setupCamera(renderWidth, renderHeight) { const camera = new PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500); camera.position.copy(new Vector3().fromArray([-1, -4, 6])); camera.lookAt(new Vector3().fromArray([0, 4, -0])); camera.up = new Vector3().fromArray([0, -1, -0.6]).normalize(); return camera; }

setupThreeScene() { const threeScene = new Scene(); const boxColor = 0xBBBBBB; const boxGeometry = new BoxGeometry(2, 2, 2); const boxMesh = new Mesh(boxGeometry, new MeshBasicMaterial({'color': boxColor})); threeScene.add(boxMesh); boxMesh.position.set(3, 2, 2); return threeScene; }

setupControls(camera, renderer) { const controls = new GaussianSplats3D.OrbitControls(camera, renderer.domElement); controls.rotateSpeed = 0.5; controls.maxPolarAngle = Math.PI * .75; controls.minPolarAngle = 0.1; controls.enableDamping = true; controls.dampingFactor = 0.05; return controls; }

componentDidMount() { this.init();

window.addEventListener( 'resize', this.onWindowResize, false );

}

componentWillUpdate(nextProps) {

}

componentDidUpdate(){

}

render() {

return (

  <div id="3dcontainer">

  <div id="GSContainer">

    </div>

  </div>

);

} }

export default GaussianSplatComp; `

cyrilpreiss commented 3 months ago

pbm

cyrilpreiss commented 3 months ago

Hi @mkkellogg i hope you'll find some time to have a look at my issue, you library is great and it's very frustrating not being able to use it.

mkkellogg commented 3 months ago

I hope I do too :) I will let you know when I have figured something out.

mkkellogg commented 3 months ago

Are there any errors in the debug console? I have seen this situation before when dealing with CORS issues (there's more about that in the README). You can also quickly check if it's a CORS issue by setting the sharedMemoryForWorkers constructor parameter to false:

this.viewer = new GaussianSplats3D.DropInViewer({
    `sharedMemoryForWorkers`: false
});
cyrilpreiss commented 3 months ago

indeed i had this error: Uncaught (in promise) DOMException: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': SharedArrayBuffer transfer requires self.crossOriginIsolated. at blob:http://localhost:3000/9df051eb-797e-4731-88fd-e2d957f8e513:134:16

with the modification you suggested it's working. can i leave this like this or should i solve the cors issue ?

mkkellogg commented 3 months ago

It's really up to you, using shared memory for the sorting web worker should boost performance, so it might be worth it. What web server are you using?

cyrilpreiss commented 3 months ago

i'm using firebase. here is a link to test: www.preiss.fr

i have two other issues: "optimizing splats" never stops Left click to set the focal point doesn't work

cyrilpreiss commented 3 months ago

actually none of your controls functions work. is it because i'm using the dropin version ?

i tried to use the not dropin version but when the viewer starts nothing shows. How to tell the view the domelement to use ?

mkkellogg commented 3 months ago

I'm not familiar with firebase, so you'll have to consult their documentation to determine how to enable CORS functionality (if you want to use shared memory for the sorting web worker)

It looks like the "Optimizing splats" message isn't going away because another error is occurring during that phase... I can't tell what it is because the code is minified.

As for the controls not working, that could be because you're using the drop-in version. If you want to use the standard Viewer, you should be able to pass your own DOM element as the root element:

const viewer = new GaussianSplats3D.Viewer({
    'rootElement': yourDOMElement
});
cyrilpreiss commented 3 months ago

thank you so much for responding so quickly !

i'm almost there !

everything is working fine in local emulator, but once online the "optimizing splats" stays on and so the mouse click interaction doesn't work (it works fine in my local emulator)

might be related to the cors issue, i will investigate.

here is the new version with the garden scene:

https://www.preiss.fr/gs/garden

cyrilpreiss commented 3 months ago

neanwhile is it possible to disable the "optimizing splats" ?

mkkellogg commented 3 months ago

The "Optimizing splats" dialog isn't going away because there was an error during the building of the splat tree: image

You could hide that dialog by passing 'showLoadingUI': false to the Viewer, but it doesn't solve the error, and the viewer won't work correctly.

cyrilpreiss commented 3 months ago

but why is there an error, i'm using your garden.ksplat it should be working. And as said it's working fine in local emulator

mkkellogg commented 3 months ago

I'm not sure why there is an error, it probably has something to do with your build & bundling system. It's impossible to tell from the debug console because the code is minified.

cyrilpreiss commented 3 months ago

it seems to come from the processSplatTreeNode funcion

mkkellogg commented 3 months ago

What version of my code are you using? Are you using the latest version?

cyrilpreiss commented 3 months ago

0.3.4 will update right now

cyrilpreiss commented 3 months ago

v0.3.5 fixed the issue ;-) thanks !!!!!

mkkellogg commented 3 months ago

Great! I'll close this for now, but let me know if you have any more questions.