search

chungyiweng / humannerf

HumanNeRF turns a monocular video of moving people into a 360 free-viewpoint video.
MIT License
791 stars 87 forks source link

Why use random background color during training #11

Closed SeanChenxy closed 2 years ago

SeanChenxy commented 2 years ago

Hi, Chungyi. Thanks for sharing the useful code. I have a question on be blow line. https://github.com/chungyiweng/humannerf/blob/7389c4049c0d5b14afdbac41a605ddad9d51dd4a/core/data/create_dataset.py#L33 I have not understood why using random background color during training: (1) If we randomize background color, we will get a high MSE loss on background pixel, but background pixel is not important. (2) If a background pixel has RGB value, a ray shooting at this pixel should has a high density. However, all the point on the ray should have zero density. Could you help me understand this design? Thanks in advance.

chungyiweng commented 2 years ago

Training a model with random bg color is for preventing the model from using the bg color to render the foreground subject.

We won't get high MSE loss on background pixels as both the rendering and ground truth use the same (randomized) bg color.

The model will be encouraged to produce zero density for background regions since if not doing so, the learned bg color with a non-zero density value in one frame will introduce errors in another frame where it has a different bg color.

SeanChenxy commented 2 years ago

Thanks for you reply, I will check it again.