Attempt to visualize 3D point cloud of ray termination - poor geometry results on real-world data

[dukeeagle]

Issue

My goal is to use NeRF to extract a 3D point cloud from my trained NeRF model on custom data. I believe I've achieved this on sample data, but I've found that my custom data that uses COLMAP for camera pose estimation gives very poor results. Is this a consequence of NeRF's poor geometry estimation with imperfect data, or is there a problem with my 3D point estimation?

Approach

I found the 3D points of all estimated ray depths while running NeRF in evaluation mode. I estimated the depth for each ray by multiplying adding the ray origin for each pixel to its direction and multiplied it by the inferred depth of each ray

p = ray origin + ray direction ray depth p: (x,y,z)*

I then collected all 3D points for each frame and combined them to visualize a single point cloud. This went well on my vanilla data, but not on my custom data.

3D point cloud of vanilla data

Here's my rendering of a point cloud of the canonical Lego bulldozer example: converts to... There's lots of noise, but this can be easily filtered out. Here's a look at the same approach to when I use custom data on a basic chair, with the exception that the camera extrinsics are fully estimated by COLMAP, unlike the perfect camera pose data in the example blender files:

3D point cloud of custom data

Summary

As you can see, NeRF has clearly failed to recreate the geometry of my scene despite the appearance being quite good in 2D form. My question is this: is there an issue with my 3D visualization approach, or is this a consequence of flawed, real-world COLMAP data? If it's the later, do you have any recommendations for improving my own data collection and processing?

[HikariS97]

Maybe the problem is the scale of depth value?

[rockywind]

How to get the real-world scale depth?

[louhz]

How to get the real-world scale depth?

I think one way is to manual setup the real-world scale depth, but the calculation standard is still unknown. LIke in Apple's ARkit, when we try to set up a new asset, we need to input the 2D image with depth and also set up the scale manually, the instruction says that we need to manually watch the reconstructed point cloud and find the best fit scale.

[szgy66]

Issue

My goal is to use NeRF to extract a 3D point cloud from my trained NeRF model on custom data. I believe I've achieved this on sample data, but I've found that my custom data that uses COLMAP for camera pose estimation gives very poor results. Is this a consequence of NeRF's poor geometry estimation with imperfect data, or is there a problem with my 3D point estimation?

Approach

I found the 3D points of all estimated ray depths while running NeRF in evaluation mode. I estimated the depth for each ray by multiplying adding the ray origin for each pixel to its direction and multiplied it by the inferred depth of each ray

p = ray origin + ray direction * ray depth p: (x,y,z)

I then collected all 3D points for each frame and combined them to visualize a single point cloud. This went well on my vanilla data, but not on my custom data.

3D point cloud of vanilla data

Here's my rendering of a point cloud of the canonical Lego bulldozer example: converts to... There's lots of noise, but this can be easily filtered out. Here's a look at the same approach to when I use custom data on a basic chair, with the exception that the camera extrinsics are fully estimated by COLMAP, unlike the perfect camera pose data in the example blender files:

3D point cloud of custom data

Summary

As you can see, NeRF has clearly failed to recreate the geometry of my scene despite the appearance being quite good in 2D form. My question is this: is there an issue with my 3D visualization approach, or is this a consequence of flawed, real-world COLMAP data? If it's the later, do you have any recommendations for improving my own data collection and processing?

Hi, I was very interested in your Lego point cloud example and recently I have been working on how to convert NeRF results from my own data into point clouds. Can you share your code. Thanks!