Question about the data split of LLFF

[Wanggcong]

Hi, I have a question about the data split of LLFF. I observed that the metrics of the 2-view setting (dsnerf, cvpr-22) are even better than other papers (e.g., 3-view of RegNeRF, cvpr-22). e.g. 20.2 psnr vs. 19.08 psnr. I have checked all of the issues about the data split but do not find the answer. The provided dataset wrap images into .npy, so it could take some time to visualize and check the results.

In RegNeRF, they split training and test sets like this way: Given an image set,

test set: for example, all=[0, 1, 2, 3, ..,19], they first take testset=all[0, 8, 16] as the test set (note, every 8 images).

training set: As for the other 17 images (k=17), others = all[1,2,3,4,5,6,7, 9,10,11,12,13,14,15, 17,18,19], they evenly sample 3 views as for training using np.linspace(0,k-1,3)=np.array([0,8,16)], say, trainingset=others[0,8,16]=all[1,9,18]

I know the data split of different papers could be different (both cvpr-22). When I implemented dsnerf using the data split method of regnerf, I found the psnr is not that high (dsnerf not better than regnerf in the 3-view setting with the same data split). I am not sure if I miss some details and I would like to double-check the data split.

Reconsider the above example, all=[0, 1, 2, 3, ..,19], what are the indices of the training set and test set?

Thank you.

[dunbar12138]

Hi, thanks for your interest!

We provide our data split here. We also provide our pretrained model here.

[PruneTruong]

Hi @Wanggcong, I have the same issue. When reproducing DS-NeRF results on 3 views using the split of RegNerF, I obtain much worse results (lower than RegNerF). Did you figure out the splits? Thanks

[Wanggcong]

Hi @PruneTruong , I think two code bases are quite different, DS-NeRF uses non-NDC to train while RegNeRF uses NDC. Also, other losses and annealing algorithms are different. I still do not know the splits since the released code provided .npy files. One way is to visualize the images and compare. But I think the splits are different. Here are what I have done.

1) I tried to implement RegNeRF upon DS-NeRF but fails, including adding annealing and NDC. I may miss the details. 2) I implemented DS-NeRF upon RegNeRF, the results slightly improve. 3) I used the data splits of RegNeRF to train DS-NeRF, the results (SSIM and PSNR) are not better than RegNeRF.

I also found that DS-NeRF adds the depth constraints on the raw depth, the depth range varies a lot in different scenes. I observed the authors may also try normalized depth or something else, but finally found the raw depth is better. Since the depth range is large, it could be sensitive if adding extra constraints on DS-NeRF.

[PruneTruong]

I see, when training and evaluation with this codebase using RegNerf splits, I obtain about 18 in psnr, 0.63 in ssim and 0.26 in lpips. Do you obtain similar results?

[Wanggcong]

Yes, I got the similar results.