mkkellogg / GaussianSplats3D

Three.js-based implementation of 3D Gaussian splatting
MIT License
1.37k stars 168 forks source link

Render SplatScenes separate #234

Closed JohannesKrueger closed 3 months ago

JohannesKrueger commented 4 months ago

To start: Great job.

My Issue: When loading multiple split scenes I have the problem that the splats from the different split scenes overlap. Loading multiple scenes also results in lower performance.

` // Load multiple splatscenes

  viewer.addSplatScenes([
      {
          'path': Path0,                                  
          'rotation': [0, 0, 0, 1],
          'scale': [0.8, 0.8, -1],
          'position': [4.28, 1.43, 6.03],
          'streamView': true,
      },
      {
          'path': Path1,                                                        
          'rotation': [0, 0, 0, 1],
          'scale': [-1, 1, -1],
          'position': [23, 1.4, 0.6],
          'streamView': true,
      },
      {
          'path': Path2,                                   
          'rotation': [0, 0, 0, 1],
          'scale': [1, 1, 1],
          'position': [9.5, 1.6, -12.4],
          'streamView': true,
      }
  ])`

Is there a way to dynamically load the splatscenes individually? I had already tried filtering the nodes by Center or ID to only render certain parts of the splatscene. However, as I understand it, the splats of all splatscenes are assigned to the nodes together, without distinguishing from which splatscene they originally come.

The rendered splats are determined in the this function based on the nodes:

gatherSceneNodesForSort = function()

i added a simple filter, to let nodes pass, based on their id's. but this doesnt work, becaused the ids have no reference to their splatscene origin.

`
if (node.id <= 8000) {

                        splatRenderCount += node.data.indexes.length;
                        nodeRenderList[nodeRenderCount] = node;
                        node.data.distanceToNode = distanceToNode;
                        nodeRenderCount++;                     

} ` This problem could be fixed by assigning the nodes (node.id) based on the their splatscene origin. But this would also cause nodes with overlapping splats if the spatscenes are to close together. Is there a way to implement this?

mkkellogg commented 4 months ago

Sorry for the late reply! Unfortunately the nodes referenced in gatherSceneNodesForSort() have no direct correlation with splat scenes. They are sections of the octree used to partition the scene for efficient raycasting and splat culling, and can contain splats from multiple scenes, so you can't filter them to hide splats from only one splat scene.

Is your goal to hide/show individual splat scenes dynamically? There's not a great way to toggle the visibility of a splat scene at runtime (maybe I should implement something like that). However you can unload a splat scene at runtime using Viewer.removeSplatScene() It's a little slow, so you probably don't want to use it frequently. At some point in a future release I will try to add functionality to hide or show splat scenes dynamically!

mkkellogg commented 3 months ago

I've made some updates in this branch that should help with dynamically toggling scene visibility: https://github.com/mkkellogg/GaussianSplats3D/tree/general_updates

You have to pass the parameter enableOptionalEffects with a value of true to the Viewer constructor and then at run time you can do:

viewer.getSplatScene(<scene index>).visible = false;
mkkellogg commented 3 months ago

Just curious if this problem is solved for you? The main branch has these updates now, so you can dynamically toggle a scene's visibility using the above syntax.

JohannesKrueger commented 3 months ago

Sorry for the late Response. I tested the updated code and this has solved the problem very well. Thank you!

mkkellogg commented 3 months ago

Awesome, glad to hear it!