search

google-research / multinerf

A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF
Apache License 2.0
3.57k stars 338 forks source link

Some doubts about the parameters #92

Open YZsZY opened 1 year ago

YZsZY commented 1 year ago

Hello author, thank you for your great work! I recently tried to make some improvements on the mipnerf360 code, I noticed that some details that are not key innovations are very different from mipnerf, and I was wondering if these factors could cause the improvement of effect.

  1. Number of resampling. I noticed that mipnerf360 resampled a total of two times, 1 resampling of proposal mlp and 1 resampling of Nerf mlp, while mip-nerf only resampled once in the fine stage, could it be that this extra resampling provides a more efficient sampling point?

  2. Encoding Dimension The mipnerf elevates the dimensionality of the input x from 3 to 2x3xL dimensions when doing the postion encoding, while the mipnerf360 encodes the dimensionality from 3 to 2x21xL (21 is the dimension of the basis P) as the diagonal matrix of covariance is no longer used. So I would like to ask author if it could be that the dimensionality of the input itself has been increased(6L->42L), so it leads to a good improvement in details and other aspects?

Sorry for disturbing you and look forward to your answers! Thanks a lot

jonbarron commented 1 year ago

1) Yes, two resampling rounds works slightly better than one round. Sorry, we should have mentioned this in the paper as another difference.

2) This is discussed in the appendix of the mipnerf360 paper, right at the beginning. Having "off-axis" encodings helps somewhat.

YZsZY commented 1 year ago

Thanks for the reply! But I'm thinking that the off-axis encoding method itself increases the input dimension of MLP by a factor of 7(3->21), so if the IPE dimension of Mip-NeRF is also improved to such an extent, will the effect be improved as well? Because I feel that the dimensional increase is really too much haha, I didn't have time to do a comparison experiment, so I wanted to ask you for advice first.

jonbarron commented 1 year ago

Depends on what strategy you use for increasing the dimensionality of the IPE in mip-NeRF. If you use the same strategy, you should expect a similar improvement.