Depth maps

[smoreira00]

In my testing procedure, I firstly ran the EndoNeRF only with the masks, and then I added the depth maps to evaluate the model's performance. I'm using a dataset whose camera poses are not fixed, unlike yours. Is it possible that, due to this, the addition of the depth maps does not produce better results than the previous case (i.e., with just the masks)? If camera poses don't have this kind of influence, what could be a possible explanation for the slight decrease of the quantitative results when adding the deph maps?

[yuehaowang]

Thanks for your good question. I think the performance decreases for these reasons:

1. Depth supervision can be regarded as a constraint in optimization. Thus, the photometric performance probably decreases, for better geometry.
2. When incorporating depth maps with moving cameras, it might be useful to disable NDC since camera origins will be shifted to a plane in NDC. Refer to: run_endonerf_helpers.py#L403-L404.
3. If camera poses are not well estimated, this will also cause blur artifacts. Without direct depth supervision, NeRF may fit misaligned areas via view-dependent encoding. However, with geometry-level supervision, the misalignment is hard to fit, resulting in some performance drop.

To highlight the effectiveness of depth, it would be a good idea to compare geometry, e.g., depth errors.

[smoreira00]

What's the difference between the flags use_depth and no_depth_sampling?

[yuehaowang]

The flag no_depth_sampling only corresponds to depth-guided ray marching. While use_depth specifies whether to load depth maps. Disabling it will turn off all depth-related modules.

[smoreira00]

When I generate my depth maps, do I have to scale them to a range between 0 and 1, or can I keep my original values in whatever range my scene is?

[yuehaowang]

You can keep the original values in the depth maps. If NDC is enabled, the depth will be normalized to [0,1].