How to initialize HyperNeRF

[jeff999955]

Hi, I am now trying to run on HyperNeRF and wondering if there are some discrepancies between the code and the paper.

In the paper, figure 2, you mentioned "The optimization process begins with Structure from Motion (SfM) points derived from COLMAP or generated randomly". However, in the code dataset_readers.py, it seems you are loading the points.npy provided by the dataset if the points3D.ply was not found.

For the sake of HyperNeRF, did you use COLMAP points or only points.npy to initialize the Gaussians given it is unclear from the code? Thank you.

[SkevyHoo]

The points.npy may generated from Colmap process in original HyperNeRF dataset.

[ingra14m]

Hi, thanks for your interest.

As @SkevyHoo said, the points.npy in HyperNeRF and NeRF-DS is generated from COLMAP. They just convert the points3D.bin in COLMAP into points.npy.

So, we can say that the real-world scene is initialized using COLMAP points.

[jeff999955]

Hi, thank you for the reply.

When I downloaded the points.npy and tried to visualize it with MeshLab, I found out the point cloud was incomplete and only containing the not moving parts of the scene. For example, in the torchocolate scene, I only see the plate in the given points.npy from HyperNeRF repo and the human figure is nowhere to be seen.

Visualized point cloud of points.npy

GT image of torchocolate

From what I discovered in Nerfies, the points.npy they provide are actually background points, where the foreground elements are removed by segmentation mask in COLMAP.

Could you clarify whether you are using the point cloud like a.) background points from points.npy or b.) reconstructed point cloud from COLMAP?

a.) background points from points.npy b.) reconstructed point cloud from COLMAP

Thank you.