Add Gaussian Splats support on the Looking Glass through WebXR

ncastaldo commented 1 month ago

I would like to submit a feature request for Gaussian Splats support on the Looking Glass through WebXR, which seems currently not available.

I have tested the library locally using the React Three Fiber library and its extension called Drei. Simple meshes are currently supported, but splats are not. This are the docs for the Splats objects in Drei: https://github.com/pmndrs/drei?tab=readme-ov-file#splat

This is the snippet to reproduce the error:

<VRButton />
<Canvas>
  <XR>
    <Splat src="https://huggingface.co/cakewalk/splat-data/resolve/main/nike.splat" />
  </XR>
</Canvas>

These are the errors from I get in the console:

WebGL: INVALID_VALUE: uniform1fv: no array
[.WebGL-0x1040678c000] GL_INVALID_OPERATION: Mismatch between texture format and sampler type (signed/unsigned/float/shadow).

I think this might be cause by the ShaderMaterial that is used inside the <Splat> component , which might not be supported by the Looking Glass WebXR library.

Here’s a link to the Drei implementation: https://github.com/pmndrs/drei/blob/master/src/core/Splat.tsx

BryanChrisBrown commented 1 month ago

interesting, I've tested this locally and it's working ok, can you let me know what version of Drei, Three.JS and the Looking Glass WebXR Library you're using?

ncastaldo commented 1 month ago

Hey Bryan, thanks for checking this out!

Yes of course, here's the version of the packages involved:

{
  "react": "18.2.0",
  "@lookingglass/webxr": "^0.6.0",  
  "@react-three/drei": "^9.108.1",
  "@react-three/fiber": "^8.16.8",
  "@react-three/xr": "^5.7.1",
  "three": "^0.164.1",
}

ncastaldo commented 1 month ago

Hey @BryanChrisBrown , can you please tell me what is your setup so that I can try to replicate and see if it works? In particular, how are you instantiating the LookingGlassWebXRPolyfill class?

Looking-Glass / looking-glass-webxr

Add Gaussian Splats support on the Looking Glass through WebXR #53