Depth map from Colmap SfM

[sandros94]

So I've noticed the existance of depth-nerfacto #1173 and I was wondering how to implement it in a workflow, but I needed more documentation and clarifications.

In my case for example I do my own Colmap camera poses, and use skip-colmap when creating the transformation.json. The datasets are all from general use cmos sensor and encoded into a video, no lidar or similar.

• Once I calculate the SfM, how can I calculate the depth map only from the point cloud? And how to export them?
• Is ns-process-data able to inject that depth map folder into the transformation.json or should it be done afterwords it's creation?
• And even more important, following the docs I see that those depth maps should be scaled in millimiters, but couldn't it be possible to have arbitrary scaling just as a reference for the lower importance supervisioning? Or all this millimiter scaling is for remaining consistent just with Polyform?

[tancik]

Once I calculate the SfM, how can I calculate the depth map only from the point cloud? And how to export them?

Given the cameras from SfM, you can iterate through each camera and project the point cloud points in the the camera frame. Keep track of the ones that intersect with the camera plane and then calculate the distance between them and the camera origin.

Is ns-process-data able to inject that depth map folder into the transformation.json or should it be done afterwords it's creation?

It isn't set up to do this at the moment. You will need to do it afterwards.

And even more important, following the docs I see that those depth maps should be scaled in millimiters, but couldn't it be possible to have arbitrary scaling just as a reference for the lower importance supervisioning? Or all this millimiter scaling is for remaining consistent just with Polyform?

You can add nerfstudio-data --depth_unit_scale_factor X to set a custom scaling factor. It defaults to 1e-3 (mm), but you can set to whatever you want.

[ManuConcepBrito]

I think this process would make sense to integrate within nerfstudio, cause it doesn't seem too obvious to me. I can take a look into implementing it if it makes sense. A couple of questions regarding the whole process:

• what's the best way of keeping track of which points intersect the image plane? Would you just apply the camera transform and the intrinsics to check which points fall within the image?
• Since the points provided by colmap are sparse, what values are given to the pixels without depth? An interpolation?

Maybe @mpmisko has some code regarding this whole process which can make depth-nerfacto way more usable for the end-user.