search

r00tman / NeRF-OSR

NeRF for Outdoor Scene Relighting [ECCV 2022]
https://4dqv.mpi-inf.mpg.de/NeRF-OSR/
235 stars 16 forks source link

What is the env params of testing or validating images? #6

Open qizipeng opened 2 years ago

qizipeng commented 2 years ago

Hi! Thanks for sharing your code and I think this is a wonderful work to solve the problems in relighting outdoor scenes by a NeRF framework. I have an issue with the env params consisting of 9*3 variables. I clearly know the network how to process the training images and optimize the env params from default values. However, I find that the network also takes the default values ​​as testing env params and do not optimize them. if i dont misunderstand. Is this reasonable? How do u get the default values?

Looking forward to your answer, Thanks again!

r00tman commented 2 years ago

Hi. Thank you for the kind words!

You can change environments used for testing with --test_env argument (implemented here). There you can either provide a folder with per-view SH environments or a path to a single SH environment written in a .txt. Even if you provide a single SH environment, you can still rotate it around the building by using --rotate_test_env argument in addition to --test_env.

Default value for the environment is taken from one of the runs of the method on our data. When we used completely random inialisation values, the model often diverged. Using these coefficients instead resulted in more consistent and better training results on the tested scenes. Coincidentally, they are also used for rendering views where no other envmap is found, which are validation and test views (when --test_env argument is not provided).

To use an external LDR/HDR envmap, you would first need to convert it to SH coefficients. The conversion will just fit the closest SH coefficients with least squares. The script for that is not yet in the repo, but I'll upload it soon, as well as the instructions on how to reproduce our numerical results from the paper. The latter involves using external environment maps and this SH conversion step too, so it should be helpful.

qizipeng commented 1 year ago

Thank you for your patience in answering! Good luck with your work

yangbaoquan commented 1 year ago

Hello, did you find where is the envmap located?

zerolfq commented 1 year ago

Can this code train a set of env parameters about the dataset

zjulabwjt commented 1 year ago

Hi. Thank you for the kind words!

You can change environments used for testing with --test_env argument (implemented here). There you can either provide a folder with per-view SH environments or a path to a single SH environment written in a .txt. Even if you provide a single SH environment, you can still rotate it around the building by using --rotate_test_env argument in addition to --test_env.

Default value for the environment is taken from one of the runs of the method on our data. When we used completely random inialisation values, the model often diverged. Using these coefficients instead resulted in more consistent and better training results on the tested scenes. Coincidentally, they are also used for rendering views where no other envmap is found, which are validation and test views (when --test_env argument is not provided).

To use an external LDR/HDR envmap, you would first need to convert it to SH coefficients. The conversion will just fit the closest SH coefficients with least squares. The script for that is not yet in the repo, but I'll upload it soon, as well as the instructions on how to reproduce our numerical results from the paper. The latter involves using external environment maps and this SH conversion step too, so it should be helpful.

@r00tman Thanks for your great work! I can't find script converting LDR/HDR envmap to SH coefficients in your repo. Can you share me this part of code ?