Extreme RAM usage, more than double compared to antimatter15/splat

[newguy123-creator]

Hi

I'm using React Three Fiber, and with mkkellogg viewer on a single .splats file I'm getting this performance: Chrome reports the site is using a staggering 1.7GB!!!

There's also some visual glitches in the mkkellogg viewer with things popping and splats appearing in front of others and weird things like that, which doesnt happen on the antimatter15 viewer. I'm adding the mkkellogg viewer like so to my App using this:

"use client";

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }) {
    const [viewer, setViewer] = useState(new THREE.Group());

    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            sharedMemoryForWorkers: false,
            showLoadingUI: true,
            // gpuAcceleratedSort: true,
        });
        const addParams = sources.map((source, position) => ({ path: source }));
        viewer.addSplatScenes(addParams, false).catch((err) => {
            console.log("Error loading splat scenes:", err);
        });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, [sources]);

    return <primitive object={viewer} />;
}

Now, compared to that, in the exact same scene with all other components and same .splat file, I use the Drei Splat wrapper, which wraps antimatter15/splat, so this instead of the mkkellogg viewer, I get slightly worse performance and my PC fans work overtime: however, in this case Chrome reports only 770mb (0.77gb), which is less than half compared to mkkellog viewer.

EDIT: Additional info: The splat was created with Postshot using the MCMC profile, then saved out, and edited with the ARAS-P Unity plugin. Then saved back out from there to a PLY file, and then converted to a .splat file using the antimatter15 dropin. The same .splat file was used in both the antimatter15 performance/ram test and the mkkellogg perfomance/ram test

[mkkellogg]

I admit, memory usage is still something I need to work on. What happens if you pass freeIntermediateSplatData: true as one of the parameters for DropInViewer ?

const viewer = new GaussianSplats3D.DropInViewer({
    freeIntermediateSplatData: true,
    sharedMemoryForWorkers: false,
    showLoadingUI: true,
    // gpuAcceleratedSort: true,
});

[newguy123-creator]

RAM goes from 1.7GB, to 1.4GB it seems. However the popping glitches where splats from the back shows in front still remain This sort of thing: correct:

...but move or rotate ever so slightly, then splats from the back suddenly comes to the front - or I dont really know what its showing, maybe inside out or something.

Then move a little left or rotate, then those blotches goes away, but as you move through the scene they pop up all over the place. For viewers that work together with three, I only know of this viewer and of the antimatter15 one, but the glitches happens only with your viewer.

[mkkellogg]

Could you share that model so I can try to debug on my end?

[mkkellogg]

As part of the next release I am going to put some serious work into addressing the excessive memory usage. As a temporary work-around you can do a couple of things:

1. You can use the progressiveLoad: true parameter for the Viewer constructor
2. You can convert your scene to a .ksplat file, which will load faster and be more memory efficient

[newguy123-creator]

As part of the next release I am going to put some serious work into addressing the excessive memory usage. As a temporary work-around you can do a couple of things:

1. You can use the `progressiveLoad: true` parameter for the `Viewer` constructor

2. You can convert your scene to a `.ksplat` file, which will load faster and be more memory efficient

Great thanks, I should be able to get back onto this in the next week and will report back on the performance. Would be good if you could have a simple online converter to ksplat, similar to how antimatter has their splat converter. You just drop in into their online demo, and it saves out a splat for you.

[mkkellogg]

Sounds good. If you want, you can also try the memory-optimizations branch. I've made some significant changes to memory usage there, and it's all detailed in this PR: https://github.com/mkkellogg/GaussianSplats3D/pull/319.

[newguy123-creator]

Sounds good. If you want, you can also try the memory-optimizations branch. I've made some significant changes to memory usage there, and it's all detailed in this PR: #319.

To install that one with NPM, do I use this one? npm install "https://github.com/mkkellogg/GaussianSplats3D.git#memory-optimizations"

Then in my splatsview component after importing: import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d"; I get this error: "Failed to resolve entry for package "@mkkellogg/gaussian-splats-3d". The package may have incorrect main/module/exports specified in its package.json."

As for converting my SH 3 PLY to a ksplat, I already use antimatter15 to convert my 630mb PLY file to a 80mb splat file with defaults. Your converter I tried defaults, except I put SH level to 2 and that results in a 184mb ksplat file. So in this case, will the ksplat still be faster and better than my splat file, even though the ksplat is larger in filesize?

[mkkellogg]

For installing the memory-optimizations branch via npm, you can try what I described here: https://github.com/mkkellogg/GaussianSplats3D/issues/324#issuecomment-2330555014

As for the larger .ksplat file, it will take longer to download because of all the extra spherical harmonics that are not in the .splat, but other parts of the loading process will be faster because it is pre-optimized.

[newguy123-creator]

I've now successfully installed and tested with the memory-optimizations branch. It does feel slightly better performing.

However I still have those visual glitches mentioned further up, where suddenly these large random splats appear over the viewport. Its almost similar to what this user describes here: https://github.com/mkkellogg/GaussianSplats3D/issues/299

However, on top of that large visual glitches, there is also micro visual glitches. I dont know how to describe it. If I keep still and dont move, these dont happen, but if I walk a bit left or look around, then its asif marching ants are walking all over the splats, almost looks like the splats are animated, or like they instantly rotate in some kind of billboard mode and instead of always facing the cam, they only jump and face cam every 10 frames or something. Dont know how else to describe it.

Again, with the antimatter15 viewer, this does not happen on the same data, although in the latest test I converted my PLY to a ksplat so I dont think antimatter can load the ksplat, but still.

There's some good news too however, well sort off. With this mkkellogg viewer, I can view my ksplat file in VR mode and it looks gloriously beautifull - as long as I dont move too much, then I see the weird micro glitches again. With the antimatter15 viewer, things look totally million times worse in VR, almost just like seethrough points and not splats, so in this instance, antimatter15 looses to mkkellogg.

I will now test the same excersise using a different PLY file as starting point, and see if I get the same visual glitches....

[mkkellogg]

If your scene has large dimensions, then setting the Viewer parameter integerBasedSort to false might help. Would you be willing to share your model so I can help debug?

[newguy123-creator]

If your scene has large dimensions, then setting the Viewer parameter integerBasedSort to false might help. Would you be willing to share your model so I can help debug?

Thanks, this seems to completely resolve the large splat artefacts that pop up over my scene.

Now just to sort out those small micro glitches....

Unfortunately I cant share the specific scene I'm busy with, but I'll see if I can train another one soon that I can send

[mkkellogg]

What are the dimensions of your scene? If they are very large, my viewer will struggle because it uses a counting sort algorithm to sort the splats, and that is famously bad for large ranges of numbers.

[newguy123-creator]

What are the dimensions of your scene? If they are very large, my viewer will struggle because it uses a counting sort algorithm to sort the splats, and that is famously bad for large ranges of numbers.

I'm checking the dimensions in Unity, just before I export to PLY, and the splatting has a scale of 2.36

Then I add a cube to see the dimensions in meter, and its relatively small. Its an interior space of a religious building A set the cube volume to cover the splat, so I can see the dimensions and it fits into a cube 171m X 96m X 197m, which in my book is relatively small, considering we usually work on much larger exterior projects, for which we will also be wanting to do splatting. In React Three Fiber, I add architectural models to the splat so everything is to a scale of 1:1. IE 10 meter scale in real life is 10m in the model.

Seeing as many people use drone footage to create their splats, these areas likely cover areas of 1km, so you might need to rethink the algorithm for sorting then...

[mkkellogg]

I just updated the memory-optimizations branch with a new option to control the size of the depth map range in the splat sort: 'splatSortDepthMapRange': 1048576.

By default the value is 'splatSortDepthMapRange': 65536. If you want to set the value larger than 1048576, you may need to set sharedMemoryForWorkers to true.

[newguy123-creator]

I just updated the memory-optimizations branch with a new option to control the size of the depth map range in the splat sort: 'splatSortDepthMapRange': 1048576.

By default the value is 'splatSortDepthMapRange': 65536. If you want to set the value larger than 1048576, you may need to set sharedMemoryForWorkers to true.

What unit is this range? Meter, centimeter? Or it has nothing to do with a distance? How would I for example detemine a good optimal value to use for a particular scene? Would it for example be possible to set this value to an auto value, and it would work this out based on the values it reads in the ksplat file, or its not so simple?

[mkkellogg]

There is no unit associated with this value, it's really a measure of the precision used to represent all the unique values of splat distances from the camera along a single camera ray (essentially). In fact a more accurate name might actually be splatSortDepthMapPrecision, and then alter the internal math a bit. The default value is 65536 which 2 ^ 16, or 16-bit. 1048576 is 2 ^ 20, or 20-bit. To get more insight into how the value is used, you can look at the source code for the sort starting here: https://github.com/mkkellogg/GaussianSplats3D/blob/2d80511077fcb83ec5cb6da7336dc38e55d4a14b/src/worker/sorter.cpp#L143 The value is (confusingly) called distanceMapRange in that part of the code :)

[mkkellogg]

So I just changed the parameter name to splatSortDistanceMapPrecision, with a default value of 16.

Would it for example be possible to set this value to an auto value, and it would work this out based on the values it reads in the ksplat file, or its not so simple?

This might be possible, but only under certain conditions: Either when a scene is not progressively loaded, or when it's progressively loaded and the file format contains the total number of splats (e.g. .ply or .ksplat). The optimal value might still be difficult to determine automatically because the layout and arrangement of those splats affects what value is ultimately needed, and because of that I think it's still a good idea to make the parameter something the user can specify.

[newguy123-creator]

So I just changed the parameter name to splatSortDistanceMapPrecision, with a default value of 16.

Would it for example be possible to set this value to an auto value, and it would work this out based on the values it reads in the ksplat file, or its not so simple?

This might be possible, but only under certain conditions: Either when a scene is not progressively loaded, or when it's progressively loaded and the file format contains the total number of splats (e.g. .ply or .ksplat). The optimal value might still be difficult to determine automatically because the layout and arrangement of those splats affects what value is ultimately needed, and because of that I think it's still a good idea to make the parameter something the user can specify.

I'm happy to report that most of the major issues seems to be resolved now. That large splat glitch I had is fixed, the micro bilboard ant matching glitch is fixed. There's a VERY MINOR glitch on distant splats now, but its so minimal, most users might not even notice it. I get 60 FPS on desktop, with the splat and bunch of other 3D glb files in the scene. On iPad Pro M2, I get around 15 FPS which is still acceptable. On my ancient android phone however, it only loads my 3d glb files at around 45 fps, but doesnt show the splat file at all. Not sure why as I dont have it hooked up currently for remote debug.

But overall, very pleased with the results and your quick responses and willingness to improve this library and help out the community. Thanks!

And did I mention it looks GLORIOUS in VR, with streaming from PC to Quest!

Is there a setting I can use, instead of loading the entire splat and only showing it when loaded, but show it in some kind of reveal where it shows bits as it loads? ...and will this harm initial loading in any way? I'm asking as it takes a few extra seconds to load on ipad pro and users might think something is wrong as nothing happens for almost 10 seconds

EDIT1: Might have spoken too soon. When I upload the build to the live site, then the splats dont load on pc, same as my android phone when viewed locally. Any ideas? And in the console I now get:

I dont get this issue locally, only when uploading to a live site

EDIT2: Here's my project json if it helps:

{
  "name": "myproject",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
    "@mkkellogg/gaussian-splats-3d": "file:../../mkkellog/GaussianSplats3D-memory-optimizations",
    "@react-three/drei": "^9.111.4",
    "@react-three/fiber": "^8.17.6",
    "@react-three/xr": "^6.2.7",
    "@vitejs/plugin-basic-ssl": "^1.1.0",
    "nipplejs": "^0.10.2",
    "r3f-perf": "^7.2.1",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "three": "^0.168.0"
  },
  "devDependencies": {
    "@eslint/js": "^9.9.0",
    "@types/react": "^18.3.3",
    "@types/react-dom": "^18.3.0",
    "@vitejs/plugin-react": "^4.3.1",
    "eslint": "^9.9.0",
    "eslint-plugin-react": "^7.35.0",
    "eslint-plugin-react-hooks": "^5.1.0-rc.0",
    "eslint-plugin-react-refresh": "^0.4.9",
    "globals": "^15.9.0",
    "vite": "^5.4.1"
  }
}

[mkkellogg]

Do you encounter this same error with version v0.4.4 of my library? Also, if you look at the network tab of the developer console, can you verify that the splat scene is actually being downloaded?

[newguy123-creator]

Do you encounter this same error with version v0.4.4 of my library? Also, if you look at the network tab of the developer console, can you verify that the splat scene is actually being downloaded?

I'v since updated the three js version in the local memory-optimizations branch to the latest, and now the splat loads, however I still get the warning about multiple three instances being imported. Again, this only happens on my live online site, not on my local dev site

EDIT: I forgot to mention, during build I also get a warning about Lottie files, which is odd as I'm not using lottie

[mkkellogg]

Yeah somehow your app is duplicating links to the three.js library, but I couldn't tell you exactly how unless I looked at your code. It might help to look at how I've used import maps in my demo pages -- that's how I ensure the version of three.js that the demo pages use and my splats library uses comes from the same file.

[newguy123-creator]

Yeah somehow your app is duplicating links to the three.js library, but I couldn't tell you exactly how unless I looked at your code. It might help to look at how I've used import maps in my demo pages -- that's how I ensure the version of three.js that the demo pages use and my splats library uses comes from the same file.

I fully installed three from npm I dont use cdn or minified links

any thoughts? If I remove the splat code, I dont get this error by the way so I suspect its something in your code. But I'll be double checking some things tomorrow to make sure...

[mkkellogg]

If you move three.js underpeerDependencies in your package.json, does that make a difference?

[newguy123-creator]

If you move three.js underpeerDependencies in your package.json, does that make a difference?

I did as you asked, here's my new package.json. It makes no difference however. I still get the issue on a live site, but locally its fine:

{
  "name": "myproject",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
    "@mkkellogg/gaussian-splats-3d": "file:../../mkkellog/GaussianSplats3D-memory-optimizations",
    "@react-three/drei": "^9.111.4",
    "@react-three/fiber": "^8.17.6",
    "@react-three/xr": "^6.2.7",
    "@vitejs/plugin-basic-ssl": "^1.1.0",
    "nipplejs": "^0.10.2",
    "r3f-perf": "^7.2.1",
    "react": "^18.3.1",
    "react-dom": "^18.3.1"
  },
  "devDependencies": {
    "@eslint/js": "^9.9.0",
    "@types/react": "^18.3.3",
    "@types/react-dom": "^18.3.0",
    "@vitejs/plugin-react": "^4.3.1",
    "eslint": "^9.9.0",
    "eslint-plugin-react": "^7.35.0",
    "eslint-plugin-react-hooks": "^5.1.0-rc.0",
    "eslint-plugin-react-refresh": "^0.4.9",
    "globals": "^15.9.0",
    "vite": "^5.4.1"
  },
  "peerDependencies": {
    "three": "^0.168.0"
}
}

Also, when looking at my network tab, the ksplat takes 8.47 seconds to load, which is pretty long. It just appears after this time and I only see my own React Three Fiber objcts before this:

So for the long loading time, I tried to remedy it by using progressiveLoad and sceneRevealMode, but it doesnt seem to do anything, the splat still just appears. Does it maybe have to do that my dropinviewer uses addSplatScenes (plural), instead of addSplatScene (single)? If so, how would I change my code to that as I tried just deleting the "s" at the end of addSplatScene"s" but then I get errors:

"use client";

import * as THREE from "three";
import React, { useState, useEffect } from "react";
import * as GaussianSplats3D from "@mkkellogg/gaussian-splats-3d";

export function SplatsView({ sources, options }) {
    const [viewer, setViewer] = useState(new THREE.Group());

    useEffect(() => {
        const viewer = new GaussianSplats3D.DropInViewer({
            freeIntermediateSplatData: true,
            sharedMemoryForWorkers: false,
            gpuAcceleratedSort: false,
            showLoadingUI: true,
            progressiveLoad: true,
            integerBasedSort: false,
            splatSortDistanceMapPrecision: 20,
            // sphericalHarmonicsDegree: 2,
            ignoreDevicePixelRatio: true,
            sceneRevealMode: "Gradual",
        });
        const addParams = sources.map((source, index) => ({
            path: source,
            // position: options[index]?.position,
            // scale: options[index]?.scale,
            // rotation: options[index]?.rotation,
        }));
        viewer.addSplatScenes(addParams, false).catch((err) => {
            console.log("Error loading splat scenes:", err);
        });

        setViewer(viewer); // This is possible since the underlying class is a THREE.Group

        return () => void viewer.dispose();
    }, []);

    return <primitive object={viewer} />;
}

Also, when building the scene, I get an issue with the above SplatsView.jsx file, which I have no idea what they are talking about...

[mkkellogg]

Well I'm a little stumped about the duplicate import issue, I think I'd have to look at your code to do any further debugging. I don't think it's specifically an issue with my library because I haven't heard of anyone else having that problem.

As for the progressive load, that currently is only supported when loading a single scene.

[newguy123-creator]

Well I'm a little stumped about the duplicate import issue, I think I'd have to look at your code to do any further debugging. I don't think it's specifically an issue with my library because I haven't heard of anyone else having that problem.

As for the progressive load, that currently is only supported when loading a single scene.

How can I adjust my dropinviewer SplatsView.jsx to use single scene? I tried simply deleting the S at the end of sceneS, but then get a bunch of errors

[mkkellogg]

addSplatScene() takes 2 arguments -- the first is the url of the scene and the second is a JSON object with options. Example:

viewer.addSplatScene(path, {
  'progressiveLoad': false,
  'splatAlphaRemovalThreshold': 10
})

[newguy123-creator]

addSplatScene() takes 2 arguments -- the first is the url of the scene and the second is a JSON object with options. Example:

viewer.addSplatScene(path, {
  'progressiveLoad': false,
  'splatAlphaRemovalThreshold': 10
})

so then in my existing SPlatsview code, instead of adding the options in the "new GaussianSplats3D.DropInViewer" I simply remove all options for the viewer, but add the options to the "addSplatScene"?

For the multiple three imports, I will check if its only the memory branch doing it, or the main branch also.

[mkkellogg]

There are two different kinds of options -- options for initializing the viewer and options for loading a scene. (these are all described in detail in the README). progressiveLoad is an option for scene loading, so it gets passed to addSplatScene(). All the other options you are passing to new GaussianSplats3D.DropInViewer are valid for initializing it.

[newguy123-creator]

Do you encounter this same error with version v0.4.4 of my library? Also, if you look at the network tab of the developer console, can you verify that the splat scene is actually being downloaded?

I tested with the main branch, and I DONT get the warning about multiple three imports

[mkkellogg]

And just to clarify, when you say the main branch, you mean you were still installing from a local source directory, not the standard npm package repository, right?

[newguy123-creator]

And just to clarify, when you say the main branch, you mean you were still installing from a local source directory, not the standard npm package repository, right?

by main branch, I mean installing via the regular way: npm install @mkkellogg/gaussian-splats-3d

[mkkellogg]

Ah, so I wonder if the problem has anything to do with the fact that you're installing the memory-optimizations directly from source. Could you try installing the main branch (v0.4.4) directly from source?

[newguy123-creator]

Ah, so I wonder if the problem has anything to do with the fact that you're installing the memory-optimizations directly from source. Could you try installing the main branch (v0.4.4) directly from source?

You're confusing me. I thought I was installing the main branch from source as here: npm install @mkkellogg/gaussian-splats-3d Is that not installing from source? This method works fine and does not give errors

The mem opt branch I followed your instructions and I had to download locally, then build it, then use npm to install that from local into my actual project like so: npm install "D:\mkkellog\GaussianSplats3D-memory-optimizations" this method results in the warning of trying to import multiple three versions

[mkkellogg]

Running the command npm install @mkkellogg/gaussian-splats-3d will install the package from the npm package repository, which is currently version v0.4.4, and does happen to correspond to what is currently in the main branch, but is not from source. Installing from source is what you're doing when you run the command npm install "D:\mkkellog\GaussianSplats3D-memory-optimizations".

When you install from source, you install whatever branch is checked out in that source directory. To install the main branch from source, you could follow the same procedure as you did for the memory optimizations branch, but use the main branch instead. You want to install from this: https://github.com/mkkellogg/GaussianSplats3D/archive/refs/heads/main.zip

[mkkellogg]

I'm going to merge the memory-optimizations and put out a new release in the next day or two. My suspicion is that if you install that version in the standard way (npm install @mkkellogg/gaussian-splats-3d), the duplicate three.js import issue might go away.

[mkkellogg]

These changes are all now in the latest release, v0.4.5, if you want to install that one directly from the npm package repository.

[mkkellogg]

Going to close this one for now, pleas let me know if you have additional questions.