The dataset with depth rendering worse than COLMAP

[TimYao18]

Hi, I tried to use online mode to capture frames and I saw it got good depth map files. But when I training the model in instant-ngp, the result always blur and not clear.

I use the same frames and remove depth files, using COLMAP to generate the transforms.json, and training, the result is much better than depth one.

Is there something I doing wrong? How can I get better result with depth files?

NerfCapture

COLMAP

[jc211]

You're probably not doing anything wrong. I think ARKit drifts and it also seems to trigger some loop closure code periodically. NeRFs are highly sensitive to bad poses. Colmap on the other hand does a global optimization, so poses from that will be better than ARKit's poses. That being said, make sure the depth supervision (which defaults to 0) in instantngp is cranked up. That helps it quite a bit. The option is a slider under "NeRF training options".

[TimYao18]

Thank you for your reply.

So, to sum up, the reason why the quality is worse is largely due to ARKit? I also tried Record3D, and it seem to have similar problems with the objects being somewhat blurry, too.

Also, if I set the Depth Supervision Strength to a value greater than 0, should I also set "enable_depth_loading" in transforms.json? I found there is "integer_depth_scale" in the transforms.json but no "enable_depth_loading"

[jc211]

"enable_depth_loading" is a new flag that I am not familiar with. Maybe they added it recently. If I had to guess it will probably default to True if its not present since it would break past behavior otherwise. The way to check if the depth is being loaded is by displaying the groundtruth depth images in the gui. If you see the depth image than they must be loaded. It also used to be that the console prints the image paths being loaded. That might have changed too.