Support for Import and Rendering of 3D Gaussian Splats

[MR-Alex42]

Is your feature request related to a problem? Please describe.

The format offers high detail including reflective, refractive, transparent, translucent, iridescent objects. I want to be able share these scans inside VR. Example

There are many cool scanning apps available to create 3D Gaussian Splats (LumAI, Poycam, etc.). Also generative AI will soon allow you to create 3D Gaussian Splats: https://cat3d.github.io/ https://zqh0253.github.io/3DitScene/

Finally there is 4D Gaussian Platting and editing coming: https://nowheretrix.github.io/HiFi4G/ https://control4darxiv.github.io/

Describe the solution you'd like

I want to be able to import and render these scans into Resonite as a new object type. Format wise they are .ply point cloud files. Resonite should support this format to stay cutting edge.

Describe alternatives you've considered

Currently the only app that allows to import 3D gaussian splats is https://www.gracia.ai/ but this is still very basic and has no multi-user capabilities. There is an Unity plugin](https://github.com/aras-p/UnityGaussianSplatting) under MIT-License that could provide some inspiration for the implementation.

Additional Context

I'm helping Martn D. (Discord-Handle) with rendering of volumetric recordings in NeosVR/Resonsite for several years now: Example https://vimeo.com/930220340/0452083fce?share=copy

Requesters

Resonite: AlienInArea51 Discord: MR-Alex mralex3078 Discord: Martn D. martnd.

[Frooxius]

I was actually looking into this, but there's a few issues right now:

• If we were to roll our own robust implementation, we'd need to invest weeks, maybe months of time into making something, which we don't have. Only option here is to use and adopt existing implentantion
• I've tried existing VR implentantion several months back, but it's been extremely heavy. 4 GB VRAM use for a single splat and would jitter heavily on RTX 3070. This isn't too good for a multi user environment. Are any new implementations more pefromant?
• It's something we'd have to eventually port to a new rendering engine, which adds a lot of extra work to the eventual switch, which is something we need to be careful about

[MR-Alex42]

I understand that the only option is to use and adopt an existing very efficient implementation. It is possible to run 3D Gaussian Splats even on a Quest 3 (XR2 Gen 2 chipset). The Gracia app already does this.

In order to reduce the burden for the Yellow-Dog-Man team I will do some research and compile a list of useful resources including serveral performance optimization papers with code that were published in the last months, hoping to increase the chance of this feature being implemented.

[Frooxius]

The papers aren't going to help us unfortunatelly, we need an open source implementation that we could adopt.

Making our own implementation based on papers would be significantly time consuming process, taking weeks, likely even months of time and that is too much to justify for this.

[MR-Alex42]

I will only look at papers that come with open source implementations (MIT-License).

[shiftyscales]

Marking as blocked by #1401 for now.

[XDelta]

As a side note, .ply point clouds are already supported for importing, just wouldn't include whatever extra data is required/included for these

[Frooxius]

Marking as blocked by #1401 for now.

This isn't specifically blocked by that, we could potentially add this before then if there's a viable open source implementation.

[Geenz]

Also more general note - so long as said implementation is easily portable to other rendering frameworks/engines (such as just needing a shader port or having minimal overhead in "connecting" it to other APIs) we can usually add porting it to the Sauce pile.

[Frooxius]

Yes that's my thinking. If there are good implementations out there, we can look at them and evaluate how good fit they would be now and how well they would port in the future and decide based on that.

[Frooxius]

Nothing this here so I don't forget, I came across this project that recently released: https://github.com/DekuLiuTesla/CityGaussian

This supports rendering really large datasets with LOD system and so on. We'd need to integrate the rendering technique, but this might be a promising approach if we pick this up.

[Frooxius]

I've started playing with gaussian splatting myself recently and learning more about them, so I'm re-evaluating this a bit x3

https://bsky.app/profile/did:plc:de3lh6vnzng6vnou2fhevu42/post/3lb6dfxybbs2q

The basic gaussian splatting rendering is actually quite simple in principle, but not the most performant. It seems that a lot of work goes into reducing their VRAM footprint and optimizing large and complex scenes.

We could start with a basic support and just have HW bruteforce it through. However the existing implementations for Unity I found have a few issues:

• Some require very new version of Unity & DX12 and specifically don't work with DX11 (which is what we use)
• Some require CUDA support, meaning AMD card users would not be able to render them out

These are some I've looked at:

• https://github.com/ptc-lexvandersluijs/Unity3DGS_VR (DX12?)
• https://github.com/ninjamode/Unity-VR-Gaussian-Splatting (Needs DX12? Maybe not? Need to poke more)
• https://github.com/clarte53/GaussianSplattingVRViewerUnity (needs CUDA)
• https://github.com/keijiro/SplatVFX (we don't use Unity VFX graph)
• https://github.com/thomas097/GaussianSplattingInUnity (looks very immature? No README)

If anybody wants to poke around these and potentially adapt them to Unity 2019.4 with DX11, I'd welcome the help. I don't know how much time I can really invest into this myself right now, since I'm busy with other stuff, but I wanted to share my findings and thoughts in this thread.

[Frooxius]

I've looked into this a bit more in my free time on the weekend.

One of the challenges with using one of the implementations above is part of the algorithm that handles sorting of the gaussians. This is done using computer shader on the GPU, using algorithms that support cross thread communication.

Problem is, Unity has support for those starting with 2020, which is newer than what we're using, meaning we cannot use those compute shaders for sorting, which complicates this.

If we were to implement this with current versions of Unity, I see two possible paths:

• Use a different GPU sort algorithm. Bitonic sort seems to be a potential alternative, that would work with current version of Unity, but likely be less efficient.
• Use CPU side sorting - this is obviously much less efficient, but we could do this potentially in a gradual and somewhat asynchronous manner - continually sorting the array in a separate thread. This could maybe work well enough, unless you change the view really fast. Most of the time once we have the data sorted, it'll already be close to sorted, so it might not even cause much of an issue. One of the implementations above doesn't even sort every frame with the GPU sort, but every 20th frame to save some performance.

[Frooxius]

Just to add a bit more, I believe the rest of the rendering process is relatively simple and feasible - it's mostly just rendering color blobs as quads and properly sampling/decoding the data (e.g. decoding a color from spherical harmonics based on the view direction).

The big part there is encoding some of the bulky data (mainly the spherical harmonics coefficients which take majority of the storage for these) into textures an encoding with stuff like BC7 to save a lot of memory - stuff we already have plumbing for - an the shader code for the decoding and such we can reuse, since it's mostly just math.