search

NVIDIAGameWorks / kaolin-wisp

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp and VQAD).
Other
1.45k stars 131 forks source link

Support for unbounded scenes #72

Open Salarios77 opened 1 year ago

Salarios77 commented 1 year ago

Hi! Is there going to be support for handling unbounded scenes via background models with re-parameterized inputs in the near future? If not, would it be possible for you to provide some tips on how to best implement that?

orperel commented 1 year ago

Hi @Salarios77 , agreed that would indeed be a useful thing to have. We should implement that, but until we do, here is a quick workaround for you, which involves extending the current tracer with one that queries some background model:

  1. Add a new forward_bg() function to your neural field (BaseNeuralField subclass). The implementation of this function should accept, i.e., a tensor of rays.

  2. Duplicate PackedRFTracer to create a customized version of your own tracer. You can find it here.

  3. Remove these lines which delete the rays tensor, as you'll need it soon as input.

  4. These lines determine the blending with the background color. Instead of assuming a fixed color (i.e. 1.0 - alpha is assumed to be multiplied with a "white" color here ), query the forward_bg function of your nef instance for a predicted background color.