Non uniform image as background

[MatteoMarengo]

Hi !

How to optimize the gaussians in the render process whilehaving a non uniform background such as an image ?

I mean should I modify this part in the cuda_rasterizer ? https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d/cuda_rasterizer/backward.cu#L530-L534

Does anyone has an idea ?

Thank you

[PanagiotisP]

Hello! That's very interesting but definitely not supported, so you would need to make your own modifications. First of all, you should pass the image to both forward and backward passes and then change the renderCuda functions in both forward.cu and backward.cu to account for the background colour. Naturally, you should have a mapping between the rays and the image. Having that, you can then replace the background colour with the value you get by querying your image with your mapping. So basically, whenever you see bg_color you should put the queried value.

If I were you, though, I would double-check the various equations that take place, to make sure everything is correct, as the original code uses black colour as background and masking any potential errors.

[MatteoMarengo]

That splats !

Thank you a lot for the detailed answer I will look at it carefully to see how it goes :)

[MatteoMarengo]

Hello,

I have tried to implement it but since then I haven't managed to have the rendered image with customized background.

Here is what I did in the train.py script:

background_image = Image.open(background_image_path).convert('RGB') transform = transforms.ToTensor() background_tensor = transform(background_image).cuda()

---

bg = bg_image if opt.random_background else bg_image render_pkg = render(viewpoint_cam, gaussians, pipe, background_tensor) # adapted render in init.py

Here is what I changed in forward.cu

if (inside) { final_T[pix_id] = T; n_contrib[pix_id] = last_contributor;

    // Fetch the background color from the image
    int bg_x = min(max(int(pix.x), 0), W - 1);
    int bg_y = min(max(int(pix.y), 0), H - 1);
    for (int ch = 0; ch < CHANNELS; ch++)
    {
        float bg_color = bg_image[ch * H * W + bg_y * W + bg_x];
        out_color[ch * H * W + pix_id] = C[ch] + T * bg_color;
    }
}

Here is what I changed in backward.cu

    float bg_dot_dpixel = 0;
        if (inside) {
            int bg_x = min(max(int(pix.x), 0), W - 1);
            int bg_y = min(max(int(pix.y), 0), H - 1);
            for (int ch = 0; ch < C; ch++)
                bg_dot_dpixel += bg_image[ch * H * W + bg_y * W + bg_x] * dL_dpixel[ch];
        }
        dL_dalpha += (-T_final / (1.f - alpha)) * bg_dot_dpixel;

Do you have any advice of what I am doing wrong ? (The background image is a RGB, 3 channels image)

Thank you so much 

[shivangi-aneja]

@MatteoMarengo Did you manage to make it work? I also want to try something like this.