kwea123
commented
3 years ago First to answer your question:
we use the neural networks to get the 4d volume
this part is slow. The rendering part is very fast.
Your idea is partly correct. You can cache the 4d data to render at different angles, but this leads to lower image quality because when you change the view angle, the rays DON'T pass exactly by the same sample points as the 4d volume that you have cached. The points' coordinates might differ just by 0.01 or a very little amount, but that makes difference! A little change in coordinate leads to a big change in color, that's why there is the embedding function. Therefore, to get better quality, we re-sample the points every time we change the angle, that's why the inference is always slow.
Back to your idea of caching the 4d volume. I have a unity project and video that realize this. It is very fast (>100FPS) in the video, but the image quality is definitely lower than if you do the "correct" inference. I didn't write python code to evaluate it, but in my opinion it might result in 5~6 points difference in PSNR.
Jiakui
On the paper , it is mentioned that inference speed of NeRF is very slow.
It seems to me that, to get an image, there are two stages. First, we use the neural networks to get the 4d volume (color and density at each 3d point position ), and then render the 4d volume to get the image. Which stage is very slow? If the first stage is very slow, it means that we can cache the 4d volueme , so that we can get images at differenct view angle using the cached data with much faster speed.
Thanks!