search

wanmeihuali / taichi_3d_gaussian_splatting

An unofficial implementation of paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering by taichi lang.
Apache License 2.0
637 stars 58 forks source link

Parameter simplifications #160

Open oliver-batchelor opened 7 months ago

oliver-batchelor commented 7 months ago

Hi there,

Thanks for your great piece of work - I'm keen to try simplifying a few things and interested in your opinion:

First one is relatively simple: There's lots of code like this where there's a parameter: in_camera_grad_color_buffer: ti.types.ndarray(ti.f32, ndim=2), # (M, 3)

Then some code later:

       point_grad_color = ti.math.vec3(
            in_camera_grad_color_buffer[idx, 0],
            in_camera_grad_color_buffer[idx, 1],
            in_camera_grad_color_buffer[idx, 2],
        )

But did you realise you can declare the parameter like this? Directly creating the vec3 instead:

in_camera_grad_color_buffer: ti.types.ndarray(ti.math.vec3, ndim=1), # (M, 3)

Second one is potentially packing parameters into vectors a little like how you did the Gaussian3D struct, so instead of having a bunch of input parameters:

    point_uv: ti.types.ndarray(ti.f32, ndim=2),  # (M, 2)
    point_in_camera: ti.types.ndarray(ti.f32, ndim=2),  # (M, 3)
    point_uv_conic: ti.types.ndarray(ti.f32, ndim=2),  # (M, 3)
    point_alpha_after_activation: ti.types.ndarray(ti.f32, ndim=1),  # (M)
    point_color: ti.types.ndarray(ti.f32, ndim=2),  # (M, 3)

They could be packed into a ti.types.ndarray(vec12, ndim=1) and unpacked into a Gaussian2D struct, a few helper abstractions can simplify it to avoid creating even more boilerplate...

Thanks! Oliver

oliver-batchelor commented 7 months ago

FWIW: Along this line I've been been making a derivative taichi_3d_gaussian_splatting (just the rasterizer for now), key idea is to split everything into parts, so that it's easy to say, replace spherical harmonics with something else. I don't think it loses much efficiency in doing so but gains a lot of flexibility.

https://github.com/uc-vision/taichi_gaussian_rasterizer