search

cmusatyalab / mega-nerf

MIT License
450 stars 54 forks source link

The validation images are also used in training #18

Closed MiZhenxing closed 1 year ago

MiZhenxing commented 2 years ago

Hi, thank you for your great open source and quick response to my previous questions. I am trying to train mega-nerf to reproduce the results. I have a question related to the validation images. As in the code below: https://github.com/cmusatyalab/mega-nerf/blob/306e06cc316dd4f5c84d0610308bcbc208228fc3/mega_nerf/runner.py#L620 It seems that the validation images are also used in training. To my understanding, it is because we need to learn the appearance embeddings of the validation images. I would to like confirm this setting and want to ask whether I understand this setting correctly. Thank you very much.

Best regards, Zhenxing Mi

hturki commented 2 years ago

Your intuition is generally correct - we use half of the validation image to learn the appearance embeddings, and then validate solely against the other half of the image.

MiZhenxing commented 2 years ago

Thank you very much. Now I understand more clearly.

MiZhenxing commented 2 years ago

Hi, sorry to reopen this issue. I still have a question about the optimization of appearance embeddings of the validation images. In NeRF in the Wild, "when evaluating error metrics on test-set images we optimize $l^{(a)}$ to match the appearance of the true image using only the left half of each image". It seems they optimize the embedding for each validation image in the validation phase. In the code of Mega-MeRF, the left half of each validation image is mixed into the training data to learn appearance embeddings of the validation images. And I didn't find codes to optimize the embedding for each validation image separately before validation. So I think the Mega-NeRF optimizes the embedding for each validation image in training phase. I just want to make sure whether you perform extra optimization for the embedding of each validation image before evaluating error metrics showing in the tables of Mega-NeRF paper. Thank you very much.

hturki commented 1 year ago

I'm not sure what it would mean for NeRF in the Wild to optimize the embedding during the validation phase vs the training phase, but I think that your understanding of Mega-NeRF is correct. We just add the left half of the images as training pixels and then evaluate on the held out right half.

MiZhenxing commented 1 year ago

Got it, thanks!