add depth prior for f2-nerf

[Bin-ze]

I added a depth prior loss to f2-nerf to constrain the wrong geometry, and gt-depth comes from sota's monocular depth estimation algorithm, but such constraints lead to more erroneous results, can you give some advice?

[jelleopard]

How to add depth map supervision in the algorithm, can you provide the code, thanks a lot

[Totoro97]

Hi, may I know how you use the depth map for supervision? Generally, the depth maps predicted by single-view depth estimation models are not suitable for direct supervision due to scale ambiguity. Maybe this work can provide a good guide on how to add depth prior for regularization, though I have not tested it yet on f2-nerf.

[Bin-ze]

Thank you very much for your reply. I have considered the scale of deep supervision and borrowed some regularization methods from other papers. My reference comes from https://localrf.github.io/localrf.pdf here,

my code snippet：

  Tensor depth_loss;
  if (dataset_->depth_exist) {
  Tensor pred_depth = disparity.clone().unsqueeze(-1);

  Tensor median_pred_depth = torch::median(pred_depth);
  Tensor var_pred_depth =  torch::var(pred_depth);
  Tensor pred_depth_norm =  (pred_depth - median_pred_depth) / var_pred_depth;

  Tensor median_gt_depth = torch::median(gt_depth);
  Tensor var_gt_depth =  torch::var(gt_depth);
  Tensor gt_depth_norm =  (gt_depth - median_gt_depth) / var_gt_depth;

  depth_loss = torch::sqrt((pred_depth_norm - gt_depth_norm).square() + 1e-4f).mean();
  }

although regularization It is possible to make the disparity of the supervision from the depth estimation and the nerf prediction be at the same scale, but I found that it still hurts the performance, the psnr with a loss rate of about 2db, I am very frustrated, I am now looking for the reason, I hope to ask for your help , because I think you are very professional, f2-nerf would be amazing if it can overcome the floater problem!

[Bin-ze]

How to add depth map supervision in the algorithm, can you provide the code, thanks a lot

My experiment is a failure, if there is progress, I will consider releasing the code

[ThomasParistech]

Have a look at the public implementation of Instant NGP that takes (potentially sparse) depth priors as input https://github.com/NVlabs/instant-ngp/issues/1238

[jelleopard]

depth= 1/disparity Tensor pred_depth = disparity.clone().unsqueeze(-1); pred_depth = 1/pred_depth;

Thank you very much for your reply. I have considered the scale of deep supervision and borrowed some regularization methods from other papers. My reference comes from https://localrf.github.io/localrf.pdf here,

my code snippet：

  Tensor depth_loss;
  if (dataset_->depth_exist) {
  Tensor pred_depth = disparity.clone().unsqueeze(-1);

  Tensor median_pred_depth = torch::median(pred_depth);
  Tensor var_pred_depth =  torch::var(pred_depth);
  Tensor pred_depth_norm =  (pred_depth - median_pred_depth) / var_pred_depth;

  Tensor median_gt_depth = torch::median(gt_depth);
  Tensor var_gt_depth =  torch::var(gt_depth);
  Tensor gt_depth_norm =  (gt_depth - median_gt_depth) / var_gt_depth;

  depth_loss = torch::sqrt((pred_depth_norm - gt_depth_norm).square() + 1e-4f).mean();
  }

although regularization It is possible to make the disparity of the supervision from the depth estimation and the nerf prediction be at the same scale, but I found that it still hurts the performance, the psnr with a loss rate of about 2db, I am very frustrated, I am now looking for the reason, I hope to ask for your help , because I think you are very professional, f2-nerf would be amazing if it can overcome the floater problem!

[Bin-ze]

depth= 1/disparity Tensor pred_depth = disparity.clone().unsqueeze(-1); pred_depth = 1/pred_depth;

Thank you very much for your reply. I have considered the scale of deep supervision and borrowed some regularization methods from other papers. My reference comes from https://localrf.github.io/localrf.pdf here, my code snippet：

  Tensor depth_loss;
  if (dataset_->depth_exist) {
  Tensor pred_depth = disparity.clone().unsqueeze(-1);

  Tensor median_pred_depth = torch::median(pred_depth);
  Tensor var_pred_depth =  torch::var(pred_depth);
  Tensor pred_depth_norm =  (pred_depth - median_pred_depth) / var_pred_depth;

  Tensor median_gt_depth = torch::median(gt_depth);
  Tensor var_gt_depth =  torch::var(gt_depth);
  Tensor gt_depth_norm =  (gt_depth - median_gt_depth) / var_gt_depth;

  depth_loss = torch::sqrt((pred_depth_norm - gt_depth_norm).square() + 1e-4f).mean();
  }

although regularization It is possible to make the disparity of the supervision from the depth estimation and the nerf prediction be at the same scale, but I found that it still hurts the performance, the psnr with a loss rate of about 2db, I am very frustrated, I am now looking for the reason, I hope to ask for your help , because I think you are very professional, f2-nerf would be amazing if it can overcome the floater problem!

Thanks for the correction, but actually I've tried to do this, but unfortunately the result will be very bad, neither anti-parallax nor depth can have good results. I doubt whether localrf's approach is correct. Although its approach comes from a 20-year-old TPAMI about depth estimation, I can't find its code implementation. I can only implement it according to the description in the paper.