mkkellogg / GaussianSplats3D

Three.js-based implementation of 3D Gaussian splatting
MIT License
1.54k stars 199 forks source link

Iphone X renders completely different with same asset and same settings #297

Closed pesa2000 closed 2 months ago

pesa2000 commented 3 months ago

Hi Mark, how are you?

I was making a couple of tests on my implementation of your viewer and i noticed something really strange happening on an old iphone of mine, on this iphoneX as you can see there is something really strange.

image

This is the render in every other devices

image

Can this be something browser related like a safari version?

mkkellogg commented 3 months ago

Have you tried different browsers besides Safari? Also, have you tried setting sharedMemoryForWorkers to false when instantiating the Viewer?

pesa2000 commented 3 months ago

image

image

these are the settings im currently using for both the scene and the viewer.

I'll let you know asap if it happens in other browsers too.

lucarinaldii commented 3 months ago

Same problem for me with iPhone 8 @mkkellogg

pesa2000 commented 3 months ago

Hi Mark, i had the possibility to test this error on another device too.

like @lucarinaldii i faced the same problem on iphone 8 and iphoneX

image

this is an example tested with both safari and chrome.

both iphones uses the latest IOS 16 version.

are there any settings that you know that can give those type of problems? maybe like "gpuAcceleratedSort" or "halfPreciosCovarianceOnGpu"?

for reference this is the correct one

image

Thank you for your time and patience.

mkkellogg commented 3 months ago

Sorry for the late response, what type of file are you trying to load? If it is a .ply file, you might want to try setting plyInMemoryCompressionLevel: 0. You should also try setting halfPrecisionCovariancesOnGpu: false (make sure to check the spelling), integerBasedSort: false, and antialiased:false (unless you know for sure it is an anti-aliased scene).

pesa2000 commented 3 months ago

Thank you for the response, ill try it asap. In the meantime i found this #265 , are these problems related?

Thank you for your time and patience.

pesa2000 commented 3 months ago

Hi Mark, like the #265 the problem seems related to the selector of the WebAssembly configuration.

image

In my case since im not using shared memory the code that is indeed executed is the first condition.

With the version 16.2 the sourceWasm is set to "SorterWasmNoSIMD" and doesnt work, i modified it(just for test) and added a new line making it that the configuration chosen is always "SorterWasmNoSIMDNonShared", in this way it seems to work.

Is there a way to map those configurations more precisely?

mkkellogg commented 3 months ago

Would you be able to try the memory-optimizations branch and see if that works for you?

pesa2000 commented 3 months ago

Tested on the same device i tested yesterday and now it seems to work, but i have a new error when loading a new splat(the first one loaded seems always to work) this is it:

image

The view completely freeze and doesnt even unload the previous one.

pesa2000 commented 3 months ago

I tested a couple more things, the error above is related to the latest version too(0.4.4), switching back to 0.4.3 fixes the issue. The memory-optimizations branch has indeed the same problem.

mkkellogg commented 2 months ago

Did you mean the memory-optimizations branch has the same error cause by runSplatSort()? I just pushed a fix for that this morning. If that issue is resolved, but the visual glitch is still there, I'll keep digging. The code should now select the SorterWasmNoSIMDNonShared WASM module if your iOS version is equal to or below 16.4, as long as enableSIMDInSort is false and sharedMemoryForWorkers is false.

pesa2000 commented 2 months ago

I mean that using the latest version(0.4.4) has now the same problem as mentioned above(the runsplatSort problem).

image

0.4.3 is fine so maybe is related to the changes above?

To reproduce this error i called the removeSplatScene from the viewer before loading a new splat, as mentioned earlier just loading one splat works correctly since i never call removeSplatScene.

mkkellogg commented 2 months ago

Good to know. I just pushed a fix for the runSplatSort() problem to the memory-optimizations branch this morning so I think it should be fixed there.

pesa2000 commented 2 months ago

Perfect, seems fixed right now for both problems.

I will try to let you know if i find more problems related to IOS devices.

Thank you very much for now, ill wait for the release.

pesa2000 commented 2 months ago

Since i have this ticket open i wanted to ask something else.

I tried to create a ShaderMaterial to use on a spherical object with a custom vertex shader, i wanted to use this sphere to wrap the splat object to create some sort of background gradient that can rotate with the camera.

When i use this sphere with a glb it renders correctly, when i try to do it on a splat it seems like that the splat is overwriting every pixel of the canvas making it impossible to render the sphere.

Is there something i can try to do? Didnt have luck with the "renderOrder" option

mkkellogg commented 2 months ago

It sounds like you essentially need to mix normal three.js objects with the splats? There's a section of the README that talks about that: https://github.com/mkkellogg/GaussianSplats3D?tab=readme-ov-file#integrating-threejs-scenes. If that's not quite what you're trying to do, maybe you can post your code and I can try to help.

pesa2000 commented 2 months ago

As soon as i have a little more time i will send to you a snippet of my code.

I'll open a new ticket for that, for now the issue above seems resolved for every IOS device. Thank you again.