Flicker in Nerfstudio-based splats

[oseiskar]

At Spectacular AI, we are using gsplat.js to display 3DGS scenes exported from Nerfstudio, with data collected and camera poses registered using our tools.

Otherwise it works great, but the rendering shows flickering splats that sometimes seem to be rendered with a wrong depth index: For example, they are visible here when rotating the scene a bit: https://www.spectacularai.com/view-splat?model=bicycle . The .splat file can be downloaded here: https://www.spectacularai.com/splats/bicycle.splat (EDIT: the code on our web page uses an older version of gsplat.js, but the issue is also present in v1.0.1)

This seems to happen on multiple browser and operating systems. It does not happen in other 3DGS renderers (e.g., Nerfstudio, antimatter15's implementation or GaussianSplats3D) so it seems to be a problem in gsplat.js. However, we don't see this with the Inria pre-trained models. Maybe this only occurrs if the splats have certain properties that do not match the assumptions in this renderer?

[dylanebert]

Are you still having this issue with the recent update (1.0.2)?

[oseiskar]

Yes. Just tested and the issue is still present with 1.0.2.

Steps to reproduce:

1. Download https://www.spectacularai.com/splats/bicycle.splat
2. Drag & drop into https://huggingface.co/spaces/dylanebert/gsplat-editor
3. (move the Bonsai model out of the way)
4. View from the pictured angle

This seems to happen with all models exported from Nerfstudio

[dylanebert]

Thanks for the details, I'm investigating the issue and will hopefully have it fixed this week!

[dylanebert]

The issue is caused by z-fighting when large splats are intersecting with the same depth.

I have pushed an update that makes the splats only re-sort when needed, so there should no longer be flickering when the view doesn't change. This should help, but it will be more difficult to eliminate z-fighting completely.

Are you able to tune parameters of NerfStudio training to encourage more densification? It seems that the example you provided has very large, elongated splats that would typically be split in the original training implementation.

[oseiskar]

Thank you for taking the time to debug. I tested 1.0.3 and now the flicker indeed eventually stops after the view is not modified.

Are you sure the main issue is z-fighting? The example below and the fact that other implementations (e.g., SuperSplat, when viewing the original PLY file) do not show flicker with the same scene suggest otherwise.

What seems to happen in the example scene in my previous message is that some of the splats get rendered in a clearly incorrect order (not always, but randomly, which causes the flicker). The geometry looks like this from another angle:

Some of the (relatively large) gray splats gets rendered in front of the rug and the bicycle wheel, even though the depth of the gray splats, viewed from the original angle, should be larger with a clear margin.

Here's another example from the same scene, where a smaller and less elongated splat also gets rendered with a wrong depth:

(there are also elongated splats near the incorrectly-rendered one but those are not the ones that are rendered incorrectly)

[dylanebert]

You're right, it does seem like something else is going wrong, likely with the depth sorting. I'll continue looking into it

[dylanebert]

I've fixed concurrency issues with depth sorting in #53

However, this issue still seems to persist. I haven't been able to figure it out - appreciate any help

[mateusz-malicki]

I think that there are two separated issues:

1. Flickering caused by z fighting: proof-no-flicker
2. wrongly calculated depth/projection (overflow?)

EDIT: in linked version there is no flickering, but exactly the same splats appears in wrong spaces

[mateusz-malicki]

Huh, I think that I've found the problem. But, it is not an overflow as i thought. I've updated demo. @oseiskar Could you please give me another model or two exported from Nerfstudio? Preferably bigger one.

[oseiskar]

Sure! Here are two other models: https://drive.google.com/drive/folders/1RxqtMUjlwfuQAxNhbnr7fP8q9f9H_XP7 The "robot" model used newer versions of Nerfstudio & Spectacular AI SDK and the wrong depth issue is a lot less prominent there (but not fully fixed if you look very closely).

[murcje]

I am having this issue also, in all my splats, using Inria and splatfacto. See https://www.mediamarc.nl/nerf/gsplat_js/?url=../splat/annabouw.sf.nocull.splat if you need a big one to test out. Cheers!

[mateusz-malicki]

@murcje Thanks for your model! Can you verify if splats in wrong order are still visible here? This version is entering low quality sorting/rendering mode when camera moves. Can you tell me if you see any noticeable improvement in performance (for the price of quality) when camera is moving/rotating compared to your current renderer? (on my machine improvement in sorting performance is kinda huge, but it is mid-tier laptop with integreted gpu - intel 11th gen i5)

[murcje]

@mateusz-malicki I checked and there is no depth issue as far as I can see, nice work!. Improvement in performance is hard for me to notice. Here is another implementation from mkkellogg, that also does not suffer from the depth issue. Performance is capped at 60 fps and that is what I observe (press i for stats)