`warpprb` only works properly when there is a triangle mesh in the scene

[gerwang]

I want to use the warpprb config instead of warp in a multiview setting, as it can model indirect illumination effects. However, it does not work properly when the scene only contains an SDF and no mesh. Strangely, it works properly when the scene contains a mesh, even if it is far away and not seen by any camera. Since adding a mesh to the scene greaterly slows down the JIT compilation of the CUDA backend, I really want to fix the issue and allow optimizing a scene that only contains an SDF.

How to reproduce:

1. Seems that the warpprb config is only used on the mirror-opt scene. I confirm it works properly on that scene:

python optimize.py mirror-opt --optconfig mirror-opt-hq --config warpprb --outputdir ./outputs --llvm --force

yields

https://github.com/rgl-epfl/differentiable-sdf-rendering/assets/25758401/ce0b8bd0-19c5-4942-9c42-2e18dd3ae14f

2. Changing the --optconfig from mirror-opt-hq from no-tex-12 also works:

python optimize.py mirror-opt --optconfig no-tex-12 --config warpprb --outputdir ./outputs --llvm --force

yields

https://github.com/rgl-epfl/differentiable-sdf-rendering/assets/25758401/3cdc11ae-d1bc-499e-853a-27ae7df73647

Note that three views are completely blocked by the wall.

3. Removing the <shape type="rectangle"> meshes in mirror-opt.xml (scene file at mirror-opt-2.zip) however, does not work properly

python optimize.py mirror-opt-2 --optconfig no-tex-12 --config warpprb --outputdir ./outputs --llvm --force

yields

https://github.com/rgl-epfl/differentiable-sdf-rendering/assets/25758401/2da82af6-6fca-4f12-8fb2-448ce114669c

4. Finally, if I do not remove the mesh shapes but put them at very high positions, it works properly even if no camera can see the rectangle shapes (scene file at mirror-opt-3.zip).

python optimize.py mirror-opt-3 --optconfig no-tex-12 --config warpprb --outputdir ./outputs --llvm --force

yields

https://github.com/rgl-epfl/differentiable-sdf-rendering/assets/25758401/7027ac78-8c46-4fb9-bc5e-c79ad22a49ac

Note

Some videos are shorter than others because I often encounter a crash

Assertion failed in /project/ext/drjit-core/ext/nanothread/src/queue.cpp:354: remain == 1

Thus, the optimization stops before running for full iterations.

[dvicini]

Hi,

Thanks for the detailed bug report. I am not surprised that the warpprb is not so robust to many different scene configurations, this integrator was only used for this particular result in the paper.

I unfortunately don't have time right now to properly look at this. I would suggest to see if the integrator base class does something different if a mesh is together. There should be some logic to switch between using optix (when meshes are present) and the simple case when we only have an SDF.

Otherwise, you can also try using an LLVM variant, as the LLVM compiler is a lot faster than the Optix compiler.

For the last error (remain==1): are you using the latest Mitsuba/drjit? If not, can you try updating to their latest releases?

[gerwang]

Thanks for your reply! I'm looking into the base ReparamIntegrator class for details.

For the last error (remain==1), I confirm that I am using the lastest pypi prebuilt mitsuba, drjit and fastsweep on Python 3.10.11 on Linux. Should I post an issue to the Mitsuba3 repo?

 ✗ pip list                                                                                           
Package         Version
--------------- --------
...
drjit           0.4.2
fastsweep       0.1.2
mitsuba         3.3.0
...

[dvicini]

Posting an issue on the Mitsuba issue tracker only really makes sense if you can reproduce the problem in a more minimal example. Otherwise it would just be too difficult to debug. Does it also happen when running some of the tutorials?

[gerwang]

Unfortunately, I only encounter this problem when running this project, and only in --llvm mode, not cuda mode. I will try to simplify the reproduction script.

[dvicini]

I see. Yes in that case your best bet is to try to simplify the script further and further until you can isolate the problem.