Performance constrainsts (CPU/GPU?)

[Zulex]

Hi,

Thanks for creating the project!

I have been playing around and it seems that I am running against performance issues that I cannot explain, as I am not certain what component is a bottleneck. The issue I am seeing is that I experience 'stuttering'. So it is not necessarily slow in the sense of low fps, but their is clear stutter. I.e. in the biking scene, whenever I am looking to the bike this stutter happens, when looking away from it, it does not. This means that for a few frames everything is smooth (as far as 30fps is smooth) and then the stutter happens.

The machine I am using has 32GB ram, i9 12900k and RTX 3090ti and I was using a link cable to stream the screen to the Quest 3.

I am getting around 30fps in the biking scene when running from Unity, which seems like a fairly low fps given the hardware. Gpu-z is telling the GPU is used between 70-85%, so there seems to be some room left here. Task-manager shows that cpu is not above 50%, however 2 threads seem to hit the 100% quite often. So this gave me the hint that their might be a CPU bounded bottleneck. However, when running the profiler, ~90% is semaphore.waitforsignal, which to my understanding means the GPU is working on something. Also, changing the resolution does not seem to effect the frame rate that much, which to my understanding would also hint a CPU bounded bottleneck.

All in all:

1. Should I expect higher fps given the hardware?
2. If this is a hardware limitation, given the story what would your guess be on which part is the bottleneck?
3. Is anyone else experiencing this stutter?

Thanks!

[Peamon]

For now, with a 3090Ti you should not expect a better FPS, I have the same performance on this scene with the same hardware.

I think your shuttering experience (that i have too) is a problem of sync, VR application work at fixed FPS (72 or 90) and try to do reprojection when a frame is not ready. Actually, the current Cuda version of gaussian splatting, does not create a depth for the generated image, so reprojection is impossible and that create shuttering.

The wait signal, wait for cuda part that render the pov and the rendering time depend of the pov. So sometime it fit in the expected time, sometime not, and the lack of depth avoid using reprojection.

I'm actually working on a new version that create a depth image during rendering, based on this pull request that will create a smoother experience for low speed hardware.

For the size of generated texture, i also have been surprised that final texture resolution, does not affect render time alot, i think it's a limitation of the original cuda rasterizer or it comes with the gaussian splatting algorithm.

I will check anyway if there is a CPU bound that i didn't see, as you said that you have 100% cpu some time, thanks for reporting.

[Orwlit]

For now, with a 3090Ti you should not expect a better FPS, I have the same performance on this scene with the same hardware.

I think your shuttering experience (that i have too) is a problem of sync, VR application work at fixed FPS (72 or 90) and try to do reprojection when a frame is not ready. Actually, the current Cuda version of gaussian splatting, does not create a depth for the generated image, so reprojection is impossible and that create shuttering.

The wait signal, wait for cuda part that render the pov and the rendering time depend of the pov. So sometime it fit in the expected time, sometime not, and the lack of depth avoid using reprojection.

I'm actually working on a new version that create a depth image during rendering, based on this pull request that will create a smoother experience for low speed hardware.

For the size of generated texture, i also have been surprised that final texture resolution, does not affect render time alot, i think it's a limitation of the original cuda rasterizer or it comes with the gaussian splatting algorithm.

I will check anyway if there is a CPU bound that i didn't see, as you said that you have 100% cpu some time, thanks for reporting.

Regarding the issue of depth rendering, I've recently been exploring a project that focuses on reconstructing scenes using sparse datasets.

This is their official repository. Building upon the original Gaussian splatting approach, they have implemented a version of diff-gaussian-rasterization that includes the capability of depth rendering.

I ran this project on my PC (rtx4090 + intel 14900), but I feel somewhat uncomfortable with it. I didn't check the exact frame rate, but I estimate it to be somewhere between 60 and 80. At the same time, I feel that the frame rate is not stable, resulting in 'flickering' phenomena. Are these issues all due to the lack of depth rendering? After all, the rendering frame rate for Gaussian splatting should be at least 144 fps on a PC.

Again this gaussian splatting VR viewer is so cool and I'd like to help, but I'm not experienced in Windows development. However, I am familiar with projects related to Gaussian splatting.

[Orwlit]

I don't know, the frame rate maybe less than 60 on 4090

[Peamon]

Hi, I'm working on the depth rendering capacity.

Regarding your performance issues you have to take into account 3 factors, the resolution, the fov and the multi POV.

For the resolution : You have a quest 3 and so, have a base resolution at 2064x2208 (4.5k pixels per eye). To help the render speed we use a scale factor of 0.5 (by default) in the texture used to render splat, so we use 2 textures of 1032x1104 (1.2k per eye). In the original gaussian splatting paper the resolution is 1920x1080 (2k pixels). We have 10% pixel more to render, it's a bit more but not explain the fps difference.

For the FOV: The first one is that the FOV of rendering is 90° instead of 60° so there is a lot more splat per pixel to render. It can explain a part of the difference, you can mesure this impact using the scene GaussianSplattingCam and changing the fov of the camera.

For the Multi POV: We have 2 renders to do, one per eye, due to gaussian splatting algorithm, two 1000x1000 rendering is not as speed as one 2000x2000 rendering because the most time passed in rendering is probably during color compilation and splat sort that are view depent. In other term, the render speed versus render resolution is not linear because it depend on the number of splat seen. I think that is the most limiting factor.

For all this reason i think your rendering speed is correct regarding your device. If you want to have a better experience, you should work on reducing the number of splat in the model you are rendering, like proposed in issue https://github.com/clarte53/GaussianSplattingVRViewerUnity/issues/9. If you have a model with less splat it will really speed up the rendering. It will be more efficent.

Best regards.

[Peamon]

The new version v1.2 has fixed the main shuttering issue. It was because the position was update during update instead of OnPreRender.

If the pointcloud is heavy you can still have experience issue due to a <72 or <90 framerate (depending of the headset).