Question: Why is `usampler2d` being used for `centersColorsTexture` instead of `sampler2d`?

[xiasun]

Hi Mark,

I am a little bit confused about why the SplatMesh is using usampler2d for the centersColorsTexture. Since the xyz coordinates of the centers need to be parsed from uint to float, if we use sampler2d, the xyz values do not need conversion, only the color needs to be converted.

Is there any specific reason for choosing usampler2d?

Many thanks!

[mkkellogg]

It's been so long since I coded that part of the viewer that I don't remember the exact details around why I made that decision, but I vaguely recall running into weird problems using a floating point texture with 4 floats packed into a texel (RGBAFormat). For some reason I could not create large textures of that format, so I couldn't support larger scenes. But with an integer based texture, I could create larger textures. That may have all been some weird issue with my own system, it didn't really make much sense to me. At any rate, there's no real performance penalty to the way I have it implemented now (at least for rendering) because it's not performing a conversion from int to float; it's floating point the whole time and it just "re-interprets" the bits in the shader as floating point.

[xiasun]

I have not encountered any texture size issue, but It seems that using usampler2d is indeed necessary. I have noticed a problem where the float-encoded integer value might lead to undefined behavior when passed to the GPU. In a locally version of Babylon 3DGS, which utilizes similar data structures to this project, I encountered a strange bug. The difference is that I used a sampler2d to store the CentersColors data.

The problem arises when certain integer values (colors) are treated as NaN (Not a Number) according to IEEE floating point standards. This type of data accounts for less than 1% of all splats and is treated as zero on Windows and M1-based Macs, resulting in no visual effects. However, on Intel-based Macs, these values are interpreted as a strange blue color. By changing the sampler2d to usampler2d, this problem was resolved.

I believe that the NaN floating point values cannot be properly passed to the GPU memory or may not be correctly interpreted with the floatBitsToUint function. Conversely, there are no NaN values for integers, so encoding floats as integers will always be safe.

There's a simple code snippet demonstrating what I just described https://jsfiddle.net/nq42px6g/. In this example, the color (188,172,166,127) is converted to the integer value 2141629628, which then becomes a NaN float. When converted back to an integer, it becomes a different value, 2145823932. I suspect there is something similar happening on the processes on GPU.

[mkkellogg]

Thank you for the detailed investigation and reporting! I think it actually makes some sense that internally the drivers or the GPU itself are making some assumptions about the texture data based on its format and taking actions based on the actual values (maybe as an optimization). It's good to know that specifying an integer format will prevent those kinds of actions from being taken, for this project and any other graphics projects I might work on :)