search

Looking-Glass / looking-glass-webxr

WebXR for Looking Glass displays
https://docs.lookingglassfactory.com/developer-tools/webxr
Apache License 2.0
65 stars 16 forks source link

Add Gaussian Splats support on the Looking Glass through WebXR #53

Open ncastaldo opened 3 months ago

ncastaldo commented 3 months ago

I would like to submit a feature request for Gaussian Splats support on the Looking Glass through WebXR, which seems currently not available.

I have tested the library locally using the React Three Fiber library and its extension called Drei. Simple meshes are currently supported, but splats are not. This are the docs for the Splats objects in Drei: https://github.com/pmndrs/drei?tab=readme-ov-file#splat

This is the snippet to reproduce the error:

<VRButton />
<Canvas>
  <XR>
    <Splat src="https://huggingface.co/cakewalk/splat-data/resolve/main/nike.splat" />
  </XR>
</Canvas>

These are the errors from I get in the console:

WebGL: INVALID_VALUE: uniform1fv: no array
[.WebGL-0x1040678c000] GL_INVALID_OPERATION: Mismatch between texture format and sampler type (signed/unsigned/float/shadow).

I think this might be cause by the ShaderMaterial that is used inside the <Splat> component , which might not be supported by the Looking Glass WebXR library.

Here’s a link to the Drei implementation: https://github.com/pmndrs/drei/blob/master/src/core/Splat.tsx

BryanChrisBrown commented 3 months ago

interesting, I've tested this locally and it's working ok, can you let me know what version of Drei, Three.JS and the Looking Glass WebXR Library you're using?

ncastaldo commented 3 months ago

Hey Bryan, thanks for checking this out!

Yes of course, here's the version of the packages involved:

{
  "react": "18.2.0",
  "@lookingglass/webxr": "^0.6.0",  
  "@react-three/drei": "^9.108.1",
  "@react-three/fiber": "^8.16.8",
  "@react-three/xr": "^5.7.1",
  "three": "^0.164.1",
}
ncastaldo commented 3 months ago

Hey @BryanChrisBrown , can you please tell me what is your setup so that I can try to replicate and see if it works? In particular, how are you instantiating the LookingGlassWebXRPolyfill class?

BryanChrisBrown commented 1 week ago

@ncastaldo one thing you'll want to be careful of with react-three-fiber is re-rendering the canvas element, I'd be cautious to put any useState hooks that'd cause the canvas itself to get re-rendered.

Are you able to share a code sandbox or other link to your project?