search

bmild / nerf

Code release for NeRF (Neural Radiance Fields)
http://tancik.com/nerf
MIT License
9.58k stars 1.34k forks source link

Why using two networks( coarse and fine)? #134

Open wzpscott opened 2 years ago

wzpscott commented 2 years ago

In the paper, you optimize two networks for hierarchical sampling. I wonder whether it is possible to use just one network and perform both uniform and importance sampling on it. Can you provide some insight on it?

kwea123 commented 2 years ago

You can read their follow-up work mip-nerf where only one network is used. It performs even better than nerf.

wzpscott commented 2 years ago

You can read their follow-up work mip-nerf where only one network is used. It performs even better than nerf.

Thank you for your quick reply. Mip-NeRF states that

This use of conical frustums and IPE features also allows us to reduce NeRF’s two separate “coarse” and “fine” MLPs into a single multiscale MLP ...

It seems to imply that coarse and fine models are necessary for original NeRF, but it's unclear to me how the IPE features relate to sampling strategy and why it allows using a single network. And I'm confused about the motivation of using two networks as it's more intuitive for me to optimize a single network like mip-nerf.

ChikaYan commented 2 years ago

I was confused by this for a while, too. I think the reason behind it is that coarse samples tend to be sparsely located in the space and be far away from the object surface, therefore they would contain lower frequency details, while fine samples would have details with much higher frequency (think it as each coarse sample should cover a larger volume than the fine sample). As illustrated in fig.4 in mip-nerf paper, the positional encoding used by original NeRF does not distinguish such a difference in frequency, therefore two networks need to be used.