scier / MetalSplatter

Render Gaussian Splats using Metal on Apple platforms (iOS/iPhone/iPad, macOS, and visionOS)
MIT License
272 stars 35 forks source link

Some questions about α-blending #6

Open moqiyinlun opened 7 months ago

moqiyinlun commented 7 months ago

I noticed that the α-blending you implemented in the shader seems not to match the original paper very well, could this be the reason for its lower rendering quality?

scier commented 7 months ago

Could you be more specific?

moqiyinlun commented 7 months ago

fragment half4 splatFragmentShader(ColorInOut in [[stage_in]]) { half2 v = in.relativePosition; half negativeVSquared = -dot(v, v); if (negativeVSquared < -kBoundsRadiusSquared) { discard_fragment(); }

half alpha = exp(negativeVSquared) * in.color.a;
return half4(alpha * in.color.rgb, alpha);

} In this one, while the original gaussian render C++ code use // Resample using conic matrix (cf. "Surface // Splatting" by Zwicker et al., 2001) float2 xy = collected_xy[j]; float2 d = { xy.x - pixf.x, xy.y - pixf.y }; float4 con_o = collected_conic_opacity[j]; float power = -0.5f (con_o.x d.x d.x + con_o.z d.y d.y) - con_o.y d.x * d.y; if (power > 0.0f) continue;

        // Eq. (2) from 3D Gaussian splatting paper.
        // Obtain alpha by multiplying with Gaussian opacity
        // and its exponential falloff from mean.
        // Avoid numerical instabilities (see paper appendix). 
        float alpha = min(0.99f, con_o.w * exp(power));
        if (alpha < 1.0f / 255.0f)
            continue;
        float test_T = T * (1 - alpha);
        if (test_T < 0.0001f)
        {
            done = true;
            continue;
        }

        // Eq. (3) from 3D Gaussian splatting paper.
        for (int ch = 0; ch < CHANNELS; ch++)
            C[ch] += features[collected_id[j] * CHANNELS + ch] * alpha * T;

        T = test_T;

And the shader in https://github.com/laanlabs/metal-splats also write as that~ I want to ask for the difference or it is a bug?(when I add 3-degree SH coefficient in the shader, I ensure the feature is the same as the C++ renderer but I still get wrong render result)

moqiyinlun commented 7 months ago

Oh, I find the problem, maybe you use gamma correction to all splat's color,but we use alpha blending to get the final color and the final color is the one who should be applied gamma correction. Suppose we have point a,b,c , the correct color is gamma(a+b+c) = (a+b+c)^2.2 while you use (gamma(a)+gamma(b)+gamma(c)) = (a^2.2+b^2.2+c^2.2), you may use a compute shader to do the correction to the final result.

scier commented 7 months ago

This is surprising — the paper is doing blending in sRGB (compressed gamma) space rather than linear space? Are you certain of this? Blending is usually done in linear space; but of course the goal for this renderer isn’t to do things “correctly”, it’s to match the training renderer as closely as possible. Have you tried this out to see if there’s a perceptible difference?

moqiyinlun commented 7 months ago

Yes, since the training picture is in srgb spcae and we use a series of gaussians to blend the final color(srgb space), if we just do srgb2linear transform to all gaussians then we will get the (a^2.2+b^2.2+c^2.2) result. I simply implement a compute shader and create a temptexture=drawable.colorTextures[0] and convert it into linear space and then blit it to drawable.colorTexture, the verify my result but seems have some delay on rendering and I want to communicate with you~ This is your result: img_v3_029h_f612c7fa-b120-4d86-b74d-e82e614b6f8g

Here is my result: QQ截图20240411153436

If you use WeChat and are open to discussing the details through it, please feel free to add me at s863438389. I'm also looking to understand more about my rendering delays.I'd like to similarly ask you about my rendering delays, thank you very much!

scier commented 6 months ago

I tried doing the sRGB -> linear conversion after rendering (as a post process compute shader run on the result frame buffer) rather than before rendering, but wasn't able to replicate the results you post -- I can't detect any visual difference. Were your two images above both produced by MetalSplatter but just with this change, or was the second image produced by a different renderer entirely? There are a number of differences between the INRIA C++ renderer and MetalSplatter, most significantly spherical harmonics and the per-pixel depth I believe INRIA's C++ renderer does, that could (and most certainly do) result in quality differences, so if you believe the sRGB conversion order is having an effect, it's important to do an apples-to-apples comparison. If you've tried implementing this change already and can see the differences, do you have the code up on a fork I could see?

vade commented 3 months ago

I think Im able to get better matching on color and alpha by making the following changes to the MetalSplat Shader :

  float gamma = 2.2;
    out.color.rgb = pow(splat.color.rgb, half3(1.0/gamma));
    out.color.a = splat.color.a;

on line 166 for the function vertex ColorInOut splatVertexShader

this does gamma correction for the RGB component but leaves the splat alpha blending value the same for when its processed in the fragment shader. Im not entirely sure the rational for the math in the fragment shader, but this appears to be much closer to my eye to other reference renders / splat files.

the follow 3 images are Truck.ply

web reference render

image

original metalsplat w/o gamma correction in my renderer (which could maybe have its own issues)

image

post correction

image

No idea if the above is helpful!

vade commented 3 months ago

And here is the Lego for reference too

image
scier commented 3 months ago

Definitely a gamma issue here, but what you're seeing in your renderer definitely doesn't match what I'm seeing; what I see is more like the "corrected" example. Can you reproduce this incorrect-gamma in the sample app?