Open moqiyinlun opened 7 months ago
Could you be more specific?
fragment half4 splatFragmentShader(ColorInOut in [[stage_in]]) { half2 v = in.relativePosition; half negativeVSquared = -dot(v, v); if (negativeVSquared < -kBoundsRadiusSquared) { discard_fragment(); }
half alpha = exp(negativeVSquared) * in.color.a;
return half4(alpha * in.color.rgb, alpha);
} In this one, while the original gaussian render C++ code use // Resample using conic matrix (cf. "Surface // Splatting" by Zwicker et al., 2001) float2 xy = collected_xy[j]; float2 d = { xy.x - pixf.x, xy.y - pixf.y }; float4 con_o = collected_conic_opacity[j]; float power = -0.5f (con_o.x d.x d.x + con_o.z d.y d.y) - con_o.y d.x * d.y; if (power > 0.0f) continue;
// Eq. (2) from 3D Gaussian splatting paper.
// Obtain alpha by multiplying with Gaussian opacity
// and its exponential falloff from mean.
// Avoid numerical instabilities (see paper appendix).
float alpha = min(0.99f, con_o.w * exp(power));
if (alpha < 1.0f / 255.0f)
continue;
float test_T = T * (1 - alpha);
if (test_T < 0.0001f)
{
done = true;
continue;
}
// Eq. (3) from 3D Gaussian splatting paper.
for (int ch = 0; ch < CHANNELS; ch++)
C[ch] += features[collected_id[j] * CHANNELS + ch] * alpha * T;
T = test_T;
And the shader in https://github.com/laanlabs/metal-splats also write as that~ I want to ask for the difference or it is a bug?(when I add 3-degree SH coefficient in the shader, I ensure the feature is the same as the C++ renderer but I still get wrong render result)
Oh, I find the problem, maybe you use gamma correction to all splat's color,but we use alpha blending to get the final color and the final color is the one who should be applied gamma correction. Suppose we have point a,b,c , the correct color is gamma(a+b+c) = (a+b+c)^2.2 while you use (gamma(a)+gamma(b)+gamma(c)) = (a^2.2+b^2.2+c^2.2), you may use a compute shader to do the correction to the final result.
This is surprising — the paper is doing blending in sRGB (compressed gamma) space rather than linear space? Are you certain of this? Blending is usually done in linear space; but of course the goal for this renderer isn’t to do things “correctly”, it’s to match the training renderer as closely as possible. Have you tried this out to see if there’s a perceptible difference?
Yes, since the training picture is in srgb spcae and we use a series of gaussians to blend the final color(srgb space), if we just do srgb2linear transform to all gaussians then we will get the (a^2.2+b^2.2+c^2.2) result. I simply implement a compute shader and create a temptexture=drawable.colorTextures[0] and convert it into linear space and then blit it to drawable.colorTexture, the verify my result but seems have some delay on rendering and I want to communicate with you~ This is your result:
Here is my result:
If you use WeChat and are open to discussing the details through it, please feel free to add me at s863438389. I'm also looking to understand more about my rendering delays.I'd like to similarly ask you about my rendering delays, thank you very much!
I tried doing the sRGB -> linear conversion after rendering (as a post process compute shader run on the result frame buffer) rather than before rendering, but wasn't able to replicate the results you post -- I can't detect any visual difference. Were your two images above both produced by MetalSplatter but just with this change, or was the second image produced by a different renderer entirely? There are a number of differences between the INRIA C++ renderer and MetalSplatter, most significantly spherical harmonics and the per-pixel depth I believe INRIA's C++ renderer does, that could (and most certainly do) result in quality differences, so if you believe the sRGB conversion order is having an effect, it's important to do an apples-to-apples comparison. If you've tried implementing this change already and can see the differences, do you have the code up on a fork I could see?
I think Im able to get better matching on color and alpha by making the following changes to the MetalSplat Shader :
float gamma = 2.2;
out.color.rgb = pow(splat.color.rgb, half3(1.0/gamma));
out.color.a = splat.color.a;
on line 166 for the function vertex ColorInOut splatVertexShader
this does gamma correction for the RGB component but leaves the splat alpha blending value the same for when its processed in the fragment shader. Im not entirely sure the rational for the math in the fragment shader, but this appears to be much closer to my eye to other reference renders / splat files.
the follow 3 images are Truck.ply
web reference render
original metalsplat w/o gamma correction in my renderer (which could maybe have its own issues)
post correction
No idea if the above is helpful!
And here is the Lego for reference too
Definitely a gamma issue here, but what you're seeing in your renderer definitely doesn't match what I'm seeing; what I see is more like the "corrected" example. Can you reproduce this incorrect-gamma in the sample app?
I noticed that the α-blending you implemented in the shader seems not to match the original paper very well, could this be the reason for its lower rendering quality?