Can avoid the redundant sigma query?

[zeng798473532]

Hi,

I find that if using the estimator.sampling and nerfacc.volrend.rendering to get the rgb values, there are two sigma queries for the valid sampling points. Can we use the sigma value queried in the estimator.sampling to execute the volume rendering? It can avoid to query the sigma twice.

Thanks.

[liruilong940607]

That's a fantastic question.

You are right it is wasteful to query sigma twice. But it is also not trivial to avoid that. The reason is in sampling we use torch.no_grad() to disable the gradient for network query and that is crucial for efficiency. But in rendering we need to enable the gradient for optimizing the network. Also in sampling the points being queried are much more than the points being queried in rendering as many of them are being filtered out in the end of the sampling function.

So in order to avoid that, we need an implementation that:

Enables no gradients evaluation on the whole set of points and the gradients computation on a subset of points.

Also the wasteful computation is only on the small fraction of total points, with only the forward pass of the network. Not a bit concern for now but definitely if you have a way to implement it without this wasteful extra query let me know!