Gaussian Splatting VR Viewer

At SIGGRAPH 2023 the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" by Kerbl, Kopanas, Leimkühler, Drettakis has been published and features impressive rendering speeds compared to other nerf techniques.

Clarte has integrated Differential Gaussian Rasterization in a Unity Native Plugin to create a OpenXR viewer of gaussian splatting learned models. This project is the original implementation of gaussian splatting renderer written in CUDA. The plugin also works without OpenXR to render gaussian splatting in Unity.

A pre-compiled version of the OpenXR viewer for windows is available for testing. You can also compile from source or open the unity project.

Other people have implemented their own gaussian splatting renderers. For example, aras-p has implemented a renderer for Unity (UnityGaussianSplatting), and the CVLab at EPFL has implemented a WebGL renderer (gaussian-splatting-web).

In this version we've add:

• Loading multiple models simultaneously
• Gaussian can be rendered on multiple camera
• Depth mix with the 3D scene

Installation

Hardware Requirements

• MS Windows VR Ready computer
• Minimal GPU: CUDA-ready GPU with Compute Capability 7.0+ (Geforce > 2060): Check your card here
• Recommended GPU: GeForce > 4070.

Our test hardware was

• CPU: Intel Core i7-11700K
• RAM: 16Go
• GPU: NVIDIA GeForce RTX 3060 Ti
• VRAM: 8Go
• Windows 10

How to start

Download the latest version of the VR viewer release.

Connect your headset and set its runtime as the default OpenXR runtime.

Launch the executable GaussianSplattingVRViewer.exe, it will launch the VR application in OpenXR environment. A menu appear with the list of model that you can load. Load "default.ply", you should see something like this.

Enjoy gaussian splatting in VR.

Loading an other model

You can display your own model. For example, you can download the original Pre-trained Models (14 GB), generate a model with Polycam service, or train your own model using the reference toolkit

To do so, copy a point_cloud.ply file, for example the one in models/bicycle/point_cloud/iteration_30000 (in the pre-trained model zip), in the same folder as the windows executable GaussianSplattingVRViewer.exe, like in the image below.

Launch the executable GaussianSplattingVRViewer.exe, the file will appear in the menu list. You can put as many as *.ply file you want. Now you can load the point_cloud.ply file instead of the default demo model.

Enjoy gaussian splatting in VR again!

By default the rendering resolution of the splats is 50% of the headset native resolution in order to limit performance issues. That's why the images displayed may be a bit pixelized. The rendering resolution can be changed in the menu using a slider (see below).

Usage

Press one Grip button fo the VR controllers to rotate and move the scene.

Press both Grip buttons to scale the scene. A line will appear between the two controllers to materialize a ruler that can help scaling the model properly.

The Joystick on the left controller can be used to translate in the scene. For example, push the joystick forward to fly in the direction pointed by the controller.

The Joystick on the right controller can be used to turn left, right or 180°.

The Menu button on left controller is used to display a menu. In this menu, you can change the rendering resolution, watch the rendering speed and the number of total splats of the models. Here you can also view the list of models you can load and lock a model to disable the rotate and scale. A button to quit the application is also available.

Alternatively, the escape or Q keys can be pressed to quit the application.

Use the plugin or compile from source

Start by cloning the whole project with recursive submodules.

git clone git@github.com:clarte53/GaussianSplattingVRViewerUnity.git --recursive

In order to use the plugin in unity see Unity Project Readme. The precompiled dll provided can be used immediately. A c# wrapper is also included in the unity project.

Alternatively, the dll can be compiled from the source. See Plugin Compilation Readme.

Performances

On the sample "bicycle" scene, from the viewpoint depicted below, rendering 1024x1024 pixels on an NVIDIA GeForce RTX 3060 Ti gives to the following measurements:

• Official SiBr viewer / OpenGL(FOV 60°): 12ms (80 FPS)
• Unity / DirectX 11 (FOV 60°): 15ms (67 FPS)
• Unity / OpenXR (2 eyes, FOV 90°): 38ms (26 FPS)

In VR more splats are rendered because of the 90° fov instead of 60° and there is two 1024x1024 texture to render, 1 per eye.

Dependencies

A modified version of Differential Gaussian Rasterization is used as submodule.

Dependencies distributed as precompiled dlls: Eigen, gl3w, glew.