search

LansburyCH / relightable-nr

A Neural Rendering Framework for Free-Viewpoint Relighting (CVPR 2020)
Other
115 stars 25 forks source link

What is the meaning of `relightable` in Neural Rendering? #1

Closed tomguluson92 closed 3 years ago

tomguluson92 commented 4 years ago

Hi, Thanks a lot for your great work! I am curious about the meaning of relightable and its relationship with free viewpoint video?

LansburyCH commented 4 years ago

Thanks for your interest in our work. For free viewpoint video, it synthesizes novel views of a dynamic scene without changing its lighting condition. As such, the synthesized appearance is limited to be under the lighting condition of the input video sequences. On the contrary, our work can synthesize the appearance under not only novel views but also novel lighting (but currently only for static objects). This second task is often referred to as relighting, and that is where relightable comes from. Naturally, we can also carry out a combination of the two tasks, which is usually referred to as free-viewpoint relighting in previous literature. The reason for associating relightable with Neural Rendering is that current neural rendering (rerendering) methods mostly focus on the change of viewpoints, but do not address much about the change of lighting, which is also an important part.

tomguluson92 commented 4 years ago

Thanks for your interest in our work. For free viewpoint video, it synthesizes novel views of a dynamic scene without changing its lighting condition. As such, the synthesized appearance is limited to be under the lighting condition of the input video sequences. On the contrary, our work can synthesize the appearance under not only novel views but also novel lighting (but currently only for static objects). This second task is often referred to as relighting, and that is where relightable comes from. Naturally, we can also carry out a combination of the two tasks, which is usually referred to as free-viewpoint relighting in previous literature. The reason for associating relightable with Neural Rendering is that current neural rendering (rerendering) methods mostly focus on the change of viewpoints, but do not address much about the change of lighting, which is also an important part.

Thanks bro, much appreciated!

Radium98 commented 3 years ago

Can you explain what's the differences between reflect_dir_map and reflect_dir_map_cam and what's the meanings of them? I'm a little confused about your code.

Radium98 commented 3 years ago

And what neural texture means?

LansburyCH commented 3 years ago

Each pixel in reflect_dir_map is the mirrored view direction around the normal direction. reflect_dir_map is in world space while reflect_dir_map_cam is in camera space.

In a nutshell, neural texture is a bunch of learnable parameters, in the form of 2D maps. The reason it is related to traditional texture map is that we sample data from it using texture coordinates, which is the same as traditional texture mapping. The concept of neural texture comes from the paper "Deferred neural rendering: Image synthesis using neural textures", which includes more detailed description about it.