How to train depth-nerfacto using monocular depth？

[YuiNsky]

Thanks for the great work of this project， i wanna know how to use monocular estimated depth to supervise the training？ since colmap depth is too sparse

[peasant98]

I believe depth nerfacto is hooked up to DepthDataset, where if you don't provide any depth data, then ZOE (state of the art monocular depth estimation) depth images will be generated. To be honest though, I would recommend a way that I think will perform better:

1. Run colmap depth and get the sparse points
2. Run the monocular depth model on the images.
3. For each image, compute a scale factor and offset with respect to the colmap points seen in each image.
4. Multiply the depth image from the model by the scale factor and add the offset
5. Train the nerf!

[aeskandari68]

I believe depth nerfacto is hooked up to DepthDataset, where if you don't provide any depth data, then ZOE (state of the art monocular depth estimation) depth images will be generated. To be honest though, I would recommend a way that I think will perform better:

1. Run colmap depth and get the sparse points
2. Run the monocular depth model on the images.
3. For each image, compute a scale factor and offset with respect to the colmap points seen in each image.
4. Multiply the depth image from the model by the scale factor and add the offset
5. Train the nerf!

As you mentioned, without having depth information, depth-nerfacto uses ZOE to estimate the depth. However, when I run it, I encounter the following error. Any feedback?

/nerfstudio/models/depth_nerfacto.py", line 86, in get_metrics_dict
    raise ValueError(
ValueError: Forcing pseudodepth loss, but depth loss type (DepthLossType.DS_NERF) must be one of (<DepthLossType.SPARSENERF_RANKING: 3>,)

https://github.com/nerfstudio-project/nerfstudio/blob/242c23f0f067064c16c49376c02271cd1cd2303b/nerfstudio/models/depth_nerfacto.py#L79-L88

[MartinEthier]

I believe depth nerfacto is hooked up to DepthDataset, where if you don't provide any depth data, then ZOE (state of the art monocular depth estimation) depth images will be generated. To be honest though, I would recommend a way that I think will perform better:

1. Run colmap depth and get the sparse points
2. Run the monocular depth model on the images.
3. For each image, compute a scale factor and offset with respect to the colmap points seen in each image.
4. Multiply the depth image from the model by the scale factor and add the offset
5. Train the nerf!

As you mentioned, without having depth information, depth-nerfacto uses ZOE to estimate the depth. However, when I run it, I encounter the following error. Any feedback?

/nerfstudio/models/depth_nerfacto.py", line 86, in get_metrics_dict
    raise ValueError(
ValueError: Forcing pseudodepth loss, but depth loss type (DepthLossType.DS_NERF) must be one of (<DepthLossType.SPARSENERF_RANKING: 3>,)

https://github.com/nerfstudio-project/nerfstudio/blob/242c23f0f067064c16c49376c02271cd1cd2303b/nerfstudio/models/depth_nerfacto.py#L79-L88

Try setting --pipeline.model.depth-loss-type SPARSENERF_RANKING

[alancneves]

I believe depth nerfacto is hooked up to DepthDataset, where if you don't provide any depth data, then ZOE (state of the art monocular depth estimation) depth images will be generated. To be honest though, I would recommend a way that I think will perform better:

1. Run colmap depth and get the sparse points
2. Run the monocular depth model on the images.
3. For each image, compute a scale factor and offset with respect to the colmap points seen in each image.
4. Multiply the depth image from the model by the scale factor and add the offset
5. Train the nerf!

Hi @peasant98 ! I will try those steps using some code from 3DGS for Depth Regularization, such as Depth Anything V2 to get the depth maps with the run.py script. And to get scale factor and offset using make_depth_scale.py from 3DGS.

I post the results here soon!

[alancneves]

I believe depth nerfacto is hooked up to DepthDataset, where if you don't provide any depth data, then ZOE (state of the art monocular depth estimation) depth images will be generated. To be honest though, I would recommend a way that I think will perform better:

1. Run colmap depth and get the sparse points
2. Run the monocular depth model on the images.
3. For each image, compute a scale factor and offset with respect to the colmap points seen in each image.
4. Multiply the depth image from the model by the scale factor and add the offset
5. Train the nerf!

Hi @peasant98 ! I will try those steps using some code from 3DGS for Depth Regularization, such as Depth Anything V2 to get the depth maps with the run.py script. And to get scale factor and offset using make_depth_scale.py from 3DGS.

I post the results here soon!

I was able to train and get results properly. The reconstruction was as good as the traditional Nerf (on my dataset).