Support for Import and Rendering of 3D Gaussian Splats

[MR-Alex42]

Is your feature request related to a problem? Please describe.

The format offers high detail including reflective, refractive, transparent, translucent, iridescent objects. I want to be able share these scans inside VR. Example

There are many cool scanning apps available to create 3D Gaussian Splats (LumAI, Poycam, etc.). Also generative AI will soon allow you to create 3D Gaussian Splats: https://cat3d.github.io/ https://zqh0253.github.io/3DitScene/

Finally there is 4D Gaussian Platting and editing coming: https://nowheretrix.github.io/HiFi4G/ https://control4darxiv.github.io/

Describe the solution you'd like

I want to be able to import and render these scans into Resonite as a new object type. Format wise they are .ply point cloud files. Resonite should support this format to stay cutting edge.

Describe alternatives you've considered

Currently the only app that allows to import 3D gaussian splats is https://www.gracia.ai/ but this is still very basic and has no multi-user capabilities. There is an Unity plugin](https://github.com/aras-p/UnityGaussianSplatting) under MIT-License that could provide some inspiration for the implementation.

Additional Context

I'm helping Martn D. (Discord-Handle) with rendering of volumetric recordings in NeosVR/Resonsite for several years now: Example https://vimeo.com/930220340/0452083fce?share=copy

Requesters

Resonite: AlienInArea51 Discord: MR-Alex mralex3078 Discord: Martn D. martnd.

[Frooxius]

I was actually looking into this, but there's a few issues right now:

• If we were to roll our own robust implementation, we'd need to invest weeks, maybe months of time into making something, which we don't have. Only option here is to use and adopt existing implentantion
• I've tried existing VR implentantion several months back, but it's been extremely heavy. 4 GB VRAM use for a single splat and would jitter heavily on RTX 3070. This isn't too good for a multi user environment. Are any new implementations more pefromant?
• It's something we'd have to eventually port to a new rendering engine, which adds a lot of extra work to the eventual switch, which is something we need to be careful about

[MR-Alex42]

I understand that the only option is to use and adopt an existing very efficient implementation. It is possible to run 3D Gaussian Splats even on a Quest 3 (XR2 Gen 2 chipset). The Gracia app already does this.

In order to reduce the burden for the Yellow-Dog-Man team I will do some research and compile a list of useful resources including serveral performance optimization papers with code that were published in the last months, hoping to increase the chance of this feature being implemented.

[Frooxius]

The papers aren't going to help us unfortunatelly, we need an open source implementation that we could adopt.

Making our own implementation based on papers would be significantly time consuming process, taking weeks, likely even months of time and that is too much to justify for this.

[MR-Alex42]

I will only look at papers that come with open source implementations (MIT-License).

[shiftyscales]

Marking as blocked by #1401 for now.

[XDelta]

As a side note, .ply point clouds are already supported for importing, just wouldn't include whatever extra data is required/included for these

[Frooxius]

Marking as blocked by #1401 for now.

This isn't specifically blocked by that, we could potentially add this before then if there's a viable open source implementation.

[Geenz]

Also more general note - so long as said implementation is easily portable to other rendering frameworks/engines (such as just needing a shader port or having minimal overhead in "connecting" it to other APIs) we can usually add porting it to the Sauce pile.

[Frooxius]

Yes that's my thinking. If there are good implementations out there, we can look at them and evaluate how good fit they would be now and how well they would port in the future and decide based on that.