Regarding suggestions for the API interface

[gotoeasy]

Please consider incorporating additional methods into the interface to enable control over various rendering effects as needed through coding.

For example:

• viewer.changeSplatAlphaRemovalThreshold(...)
• viewer.changeRenderMode(...) - Normal mode/Point cloud mode
• viewer.removeSplatScene(...)
• ...

[gotoeasy]

Is it possible to achieve some rendering effects similar to those in the image below by adjusting certain parameters or providing interface methods? For example, tweaking the model's perspective to generate an image resembling a sketch effect that I personally like—how cool would that be!

https://github.com/graphdeco-inria/gaussian-splatting/issues/427

[mkkellogg]

Functions like viewer.changeSplatAlphaRemovalThreshold() and viewer.removeSplatScene() make sense, but I'm not sure what you mean by viewer.changeRenderMode(), specifically what do you mean by "normal mode" and "Point cloud mode" ?

As far as mimicking the effects in the above pictures, I imagine there are all kinds of cool effects I could enable if I spent some time experimenting, or looked into what others have done with GS renderers, it's really just a matter of having the time to do so :)

[gotoeasy]

Functions like viewer.changeSplatAlphaRemovalThreshold() and viewer.removeSplatScene() make sense, but I'm not sure what you mean by viewer.changeRenderMode(), specifically what do you mean by "normal mode" and "Point cloud mode" ?

"normal mode" refers to the current completed effect, while "Point cloud mode" refers to a display in a point-like manner, similar to the top left image above, or the appearance of a point model as initially shown by Luma.

The reason for this request is mainly because in large scene displays, it can provide a focusing effect (rendering areas of non-focus in a point-like manner), reduce model file sizes, or significantly improve rendering performance.

[koktavy]

+1 for removeSplatScene, could you provide pseudo code for this? I'd give it a go.

I see dispose in SceneHelper but that looks more like a total cleanup.

[mkkellogg]

Yeah dispose() will release all resources held by the viewer. I plan on adding a removeSplatScene() function in the next release, so you won't have to wait too much longer :)

[gotoeasy]

In the current version v0.3.8, pressing the 'P' key switches to PointCloudMode, but there is currently no transition effect during the switch, which can feel rather abrupt. To achieve a transition effect similar to what Luma has, how should the code be modified? @mkkellogg

[mkkellogg]

You'd have to modify the shader for SplatMesh in this area: https://github.com/mkkellogg/GaussianSplats3D/blob/5c1fd4dd2b34512bfd20ce4f88efd4bedeac570a/src/SplatMesh.js#L303 Basically you'd want to interpolate between using eigen-values calculated from the covariances and the hard-coded eigen-values for point-cloud mode (0.2) over some time interval. BTW 0.2 was chosen arbitrarily, both eigen-values just needed to be equal to each for point-cloud mode.

[gotoeasy]

Thanks for your response, after continuous experimentation I can finally see the result with splat. But it doesn't seem to work with ksplat, the values of both visibleRegionFadeStartRadius and visibleRegionRadius are always 0. Does ksplat not support the transition effect of the ever-expanding radius?

[gotoeasy]

Thanks for your response, after continuous experimentation I can finally see the result with splat. But it doesn't seem to work with ksplat, the values of both visibleRegionFadeStartRadius and visibleRegionRadius are always 0. Does ksplat not support the transition effect of the ever-expanding radius?

the ksplat converted in version 0.4.0 is ok, please ignore this.

[gotoeasy]

The current transition approach seems to prioritize the outward spread of a radius from a horizontal center point, without considering height. For example, a table would load from the base upwards, while the tabletop would load later, which can feel unnatural. Similarly, for a building, the rooftop would load later.

Would it be possible to incorporate height into the consideration, such that loading within the same radius prioritizes height?

[mkkellogg]

The reason I chose the approach to revealing the scene is that it matches the way splats are physically organized in .ksplat files, and the reason splats are organized in that manner is that it allows for better cache coherency for sorting and rendering at runtime. It would be possible to arrange the splats in a way that would make your approach to revealing the scene possible, but you'd have to implement a new way to do that as part of the export to .ksplat, and you'd need to include a flag in the file so that when it comes time to load the splats, the viewer knows how the splats are arranged and can choose the appropriate scene reveal function.

The code that performs the current splat arrangement is here: https://github.com/mkkellogg/GaussianSplats3D/blob/ec323ad69dab6cfbfbef419019d626a3dcacd1b5/src/loaders/SplatPartitioner.js#L48

You'd have to add a new function to achieve your own desired organization, and then update the relevant parts of the code to use it, as well as add the aforementioned flag specifiying organization method to the .ksplat format. Then you'd have to update SplatMesh to choose the approach reveal mode at load time, as well as implement the new reveal mode.

So it definitely wouldn't be a trivial amount of work, but it's certainly doable :)

[gotoeasy]

Considering only one scene, how to get the maximum height and set it to sceneCenter? If set in this way, I think it may be possible to display it progressively from high to low, and the file structure of the model does not need to be changed.

[gotoeasy]

Modifying sceneCenter doesn't seem to work, a compromise is to just compare the horizontal distance, the high point can be shown at the same time. Not sure if this is a good approach, so I'm not submitting a PR.

// SplatMesh.js
-        float centerDist = length(splatCenter - sceneCenter);
+        float centerDist = length(vec3(splatCenter.x, 0.0, splatCenter.z)- sceneCenter);

[gotoeasy]

Hi, @mkkellogg

For a long time, there has been an urgent need for methods to load and render large scenes efficiently.

In general, the user should programmatically determine which models need to be loaded for the current viewport and which previously loaded models need to be unloaded.

Once the models to be loaded have been identified, the user would like to call an API method that provides a progressive loading effect. For each viewport change, the target models to be loaded would be re-identified, and then the API would be called to update the display.

For example, a street might be composed of many models. As the user walks down the street, the target models within the visible range will constantly change. The user would continuously call the API with the computed target models as parameters. The new target models would be progressively loaded during download, enabling a smooth rendering effect.

This is similar to how a single model is downloaded and rendered concurrently, except that multiple smaller models are being downloaded simultaneously. These models are continuously merged, combined, and updated to render. If a model is no longer needed for rendering, it is deleted. By controlling the number or size of target models within the visible range, the browser will not crash, ultimately achieving the goal of smoothly viewing the entire street.

The illustrative code would look like this:

viewer.addSplatScene(building01, options01);
viewer.addSplatScene(building02, options02);
viewer.addSplatScene(building03, options03);
viewer.addSplatScene(building04, options04);
viewer.addSplatScene(building05, options05);

// When walking forward
viewer.removeSplatScene(pathOrIdOfBuilding01)
viewer.addSplatScene(building06, options06);

// Continue walking forward
viewer.removeSplatScene(pathOrIdOfBuilding02)
viewer.addSplatScene(building07, options07);

I'm not sure if I have clearly expressed the scene requirements. How would you consider implementing this type of scene?

[mkkellogg]

Supporting larger scenes is definitely something on the roadmap for this project, but it's very much a non-trivial thing to implement. Right now the viewer is just not set up to frequently load & unload splat scenes efficiently; it would require a significant refactor or redesign to allow it to do so. But more importantly I don't think that's necessarily the correct way to support large scenes. Ultimately I want to support an LOD-based approach like the one described in this paper: https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/. Currently they have not released the source code and they don't have any test data I can play with, so I can't really start working on anything. But as soon as they do, I plan on integrating their approach.

[lewibs]

You may find the potree repository interesting. It's extremely messy... But it's extremely good at loading large points clouds very quickly. At my company we would load 5-10GB point clouds almost instantly. If I recall correctly, it basically segments the data then uses a front end client to load in the data chunks when they are needed. You have to preprocess the data though so that is a pain.

[gotoeasy]

Frequent model data changes should affect performance, but it seems that it has to be changed. The combined .splat for large scenes is probably several GB in size, and the browser will definitely crash. Based on LOD, the data within the viewing range also needs to be changed.

This is a real challenge!

[gotoeasy]

Supporting larger scenes is definitely something on the roadmap for this project, but it's very much a non-trivial thing to implement. Right now the viewer is just not set up to frequently load & unload splat scenes efficiently; it would require a significant refactor or redesign to allow it to do so. But more importantly I don't think that's necessarily the correct way to support large scenes. Ultimately I want to support an LOD-based approach like the one described in this paper: https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/. Currently they have not released the source code and they don't have any test data I can play with, so I can't really start working on anything. But as soon as they do, I plan on integrating their approach.

The released source code is written in Python. Is it a good reference point?

[gotoeasy]

Let's close this for now.