search

dunbar12138 / DSNeRF

Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
https://www.cs.cmu.edu/~dsnerf/
MIT License
746 stars 126 forks source link

Confusion on files in split_allview_npy #78

Closed xiyichen closed 1 year ago

xiyichen commented 1 year ago

Hello, thanks for your work and code. I unpacked the .npy files provided for the fern_2view scene. The files and shapes of matrices are as follows:

poses_bounds.npy (20, 17) bds.npy (20, 2) test_images.npy (3, 756, 1008, 3) test_poses.npy (3, 3, 5) train_depths.npy (2,) keys: 'depth', 'coord', 'error' (shape of each is 2182,) train_images.npy (2, 756, 1008, 3) train_poses.npy (2, 3, 5)

Since I am trying to generate these for my own data, I am trying to understand the provided ones and I have a few questions:

  1. Why are there 20 frames in poses_bounds.npy and bds.npy, if you only use 2 images for training and 3 for testing?
  2. As mentioned in @8, only the 2 training images are supposed to be used to run COLMAP sparse reconstruction. How do I infer pose vectors for the test images?
  3. In the provided data there are only 1081 sparse 3D points for this scene. Why are there 2182 points in train_depth.npy file?
  4. As mentioned in LLFF, each 17-dimensional vector in poses_bounds.npy is composed of the (3,5) pose vector concatenated by 2 depth values for closest and farthest scene content. Is it how you generated them here? I don't see a correspondence between the values in train_poses.npy/bds.npy and poses_bounds.npy.
dunbar12138 commented 1 year ago

Thanks for your interest!

Sorry for the confusion. poses_bounds.npy is actually not useful in the dataset. And seems we include the wrong bds.npy file, but it doesn't affect much. We have updated the data.

To get these files, we re-run COLMAP and post-processed it, including re-centering the poses and filtering some 3D points from specific views.

Regarding running on your own data, you don't have to generate those files exactly. Instead, you can directly use the function here to load the depths after running COLMAP.

If you want to generate views from pre-defined test poses, you need to calculate the relative transformation between training poses and test poses and infer the test poses in the NeRF coordinate system.

Hope it helps!

Yarroudh commented 1 year ago

@dunbar12138

What do you mean by using this function? Where should we use it? in img2poses.py or any other script, I really do not understand. I tried but I'm having this error:

Mismatch between imgs 0 and poses 499 !!!!
Traceback (most recent call last):
  File ".\imgs2poses.py", line 20, in <module>
    load_colmap_depth(args.scenedir)
  File "C:\Users\Administrateur\Desktop\COLMAP-Reconstruction\DSNeRF\load_llff.py", line 350, in load_colmap_depth
    _, bds_raw, _ = _load_data(basedir, factor=factor) # factor=8 downsamples original imgs by 8x        
TypeError: cannot unpack non-iterable NoneType object