search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.83k stars 1.32k forks source link

Quadrature computation in raymarcher #639

Closed Wang-Tianyu closed 3 years ago

Wang-Tianyu commented 3 years ago

In NeRF the rendered color is computed with the quadradture Screenshot from 2021-04-12 17-24-50

where c_i is the color, sigma_i is the density and delta_i is the distance between sampled points along a ray. However, in the implementation of ImplicitRenderer, I didn't find where delta_i is actually used. Please correct me if I am wrong, but the ray_density variable in Raymarcher refers to only 1 - exp(-sigma) without the delta term.

Is this an intended behavior?

florianHofherr commented 3 years ago

If you are referring to the NeRF implementation in the pytorch3d repo, I would say, that the distance between the sampling points is included in the _get_densities function of the implicit function in NeuralRadianceField. The function seems to return 1-exp(-delta*sigma) as "densities". I was wondering about the same thing, but when combining both functions, to me it seemed that the implementation matches the formula that you have shown above.

Wang-Tianyu commented 3 years ago

If you are referring to the NeRF implementation in the pytorch3d repo, I would say, that the distance between the sampling points is included in the _get_densities function of the implicit function in NeuralRadianceField. The function seems to return 1-exp(-delta*sigma) as "densities". I was wondering about the same thing, but when combining both functions, to me it seemed that the implementation matches the formula that you have shown above.

_get_densities return 1 - (-raw_densities).exp() where raw_densities is the output of the nerual radiance field. In fact, sample distance delta_i is never an input of _get_densities. Thus I don't think they are the same. The only explanation I can think of is that uniform spaced sampling is assumed, making delta_i a constant so we can hide it from the optimization process.

florianHofherr commented 3 years ago

Maybe we mean different things. I am referring to _get_densities in this file. It takes depth_values as input and computes the deltas as second operation.

Wang-Tianyu commented 3 years ago

Maybe we mean different things. I am referring to _get_densities in this file. It takes depth_values as input and computes the deltas as second operation.

Yes, we do mean different things. I am referring to the tutorial notebook for neural radiance field as well as textured volume. In the file you are referring to, the NeRF rendering equation is implemented exactly.

davnov134 commented 3 years ago

Hi, thanks for the question. Indeed the tutorial notebook is a simplified version which does not take the depth intervals into account. As @florianHofherr mentioned, the reproduced version of NeRF does factor in the depth intervals.

The only explanation I can think of is that uniform spaced sampling is assumed, making delta_i a constant so we can hide it from the optimization process.

Yes, this is exactly the case.