The procedure of adding depth loss

[JerryPW]

Hi! Sorry to bother again! Since I'm not really familiar with CUDA programing, I'm not sure whether my process in on the right way. I noticed that you have posted a version of depth-diff-gaussian-rasterization with forward and backward of depth loss in here, so all I should do is to add my depth pic into the class Camera() in scene/cameras.py， and then call the depth in train.py when calculate loss, right?

# Render
render_pkg_re = render(viewpoint_cam, gaussians, pipe, background, d_xyz, d_rotation, d_scaling, dataset.is_6dof)
image, viewspace_point_tensor, visibility_filter, radii = render_pkg_re["render"], render_pkg_re[
    "viewspace_points"], render_pkg_re["visibility_filter"], render_pkg_re["radii"]
pred_depth = render_pkg_re["depth"]

# Loss
gt_image = viewpoint_cam.original_image.cuda()
mask = viewpoint_cam.mask
gt_depth = viewpoint_cam.depth

Ll1 = l1_loss(image, gt_image)
loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image))

if gt_depth is not None:
    gt_depth = gt_depth.cuda()
    loss += l1_loss(pred_depth[0], gt_depth)
loss.backward()

Here's the small demo that I added depth loss, and I wondered am I right, or there's other loss function should be called when calculate depth loss?

Looking foward to your reply!

[ingra14m]

Hi, your calculation process is correct.

In my experiment, I was able to achieve relatively good geometric optimization by just adding an L1 depth loss. One point to note is that the depth range from rasterization is not 0-1. You need to use some mapping, such as normalization, to output the depth.png. Here is a simple example for depth visualization

[JerryPW]

Thanks for point out the depth range problem! I haven't noticed that and it produce a terrible psnr, which bother me a long time. I'll try your suggestion in a moment.