IndexError during training & custom dataset & optimized camera poses

[hdzmtsssw]

Thanks for great work! When I train my own dataset, an error occurred.

 12%|██████▍                                             | 25000/200002 [2:44:17<19:10:03,  2.54it/s]
Traceback (most recent call last):
  File "/raid/BAD-NeRF/train.py", line 324, in <module>
    train()
  File "/raid/BAD-NeRF/train.py", line 255, in train
    mse_render = compute_img_metric(imgs_sharp, imgs_render, 'mse')
  File "/raid/BAD-NeRF/metrics.py", line 75, in compute_img_metric
    im1[i], im2[i]
IndexError: index 26 is out of bounds for axis 0 with size 26
Command exited with non-zero status 1

1. My data only include 30 common images. I just follow README.MD: put my own data in the folder images, run imgs2poses.py script. Then cp -r images/ images_1/ and cp -r images/ images_test/. Do you have any idea about this?

2. I wonder what's the meaning of the three different folders. How to create my own dataset when I only have some common images(just like the vanilla nerf_llff_data)?

3. Besides, I notice that the method can optimize camera poses, I wonder how can I get them and how can I use them. If I want to use the optimized camera poses just as the original colmap format(qvec and tvec in images.bin, or W2C / C2W), what should I do for transformation?

[wangpeng000]

Hello,

Q1, what you have done is right and you'll get a poses_bounds.npy file.

Q2, images folder only contains the blurry images (30 for you); images_1 contains not only the blurry images (the same as folder "images"), but also some sharp novel view images for novel view syhthesis evaluation (if you don't have sharp novel view inages, then images_1 is the same as images). images_test is the ground truth images corresponding to blurry images, which are used for computing metrics.

If you only have some common (you mean blurry?) images, first put them to images_1 and images_test folders and then set the parameter novel_view in configs.txt to False. Of course, the metrics computed by the images_test is wrong because these images are not ground truth, but it is the easiest way to make the code ready.

And I will add the Q2 answer to the README.md file.

Q3, I think the variable all_poses is what you need.

[hdzmtsssw]

Thanks for your fast reply. I have successfully trained my own data! However, I have encountered a couple of minor issues.

Q1: I am still a bit confused about the blurry images. If all my data is clear (maybe sharp?), can I still use them as proxies for blurry images to optimize the camera poses?

Q2: I have successfully obtained the cam_poses from poses_render.txt (I believe they are the c2w poses), but the visualization is inconsistent with the ground truth. I suspect that there may be some transformations involved. How can I transform these poses to align them with the colmap style?

Camera poses with ids greater than 30 are ground-truth poses, while those with ids less than 30 are the optimized poses.

[wangpeng000]

Q1, theoretically BAD-NeRF can jointly optimize NeRF with sharp or blurry images. If the images are sharp, then the optimized start pose and end pose are very close. Howerve, I didn't run this experiment and you can try it.

Q2, we adopt the recenter_poses function in orginal nerf-pytorch. Therefore, the pose_bounds.npy poses are transfered to a new coordinate and I don't know how to inverse this process. In our paper, we use the EVO Toolbox to align the poses_render.txt and the groundtruth poses. Note that there is a scale factor between poses_render.txt and groundtruth poses.