Is mitsuba.OptixDenoiser differentiable?

[saedrna]

As seen in the tutorial, mitsuba 3 has a wrapper for the Optix AI Denoiser. If the denoiser differentialble given the code as in the tutorial?

noisy = mi.render(scene, spp=1)
# Denoise the rendered image
denoiser = mi.OptixDenoiser(input_size=noisy.shape[:2], albedo=False, normals=False, temporal=False)
denoised = denoiser(noisy)

I see that the work in nvdiffrecmc also use such denoiser, and it seems to be differentialble, so we can use small spp to save some memory and time.

[njroussel]

Hi @saedrna

No it is not, at least to my knowledge. It is just a black box. What I'm guessing that these papers/methods do, is that they do some AD surgery to compute their loss on the denoised output, but propagate gradients on the original image. You can do this in Mitsuba too:

import mitsuba as mi
import drjit as dr

mi.set_variant('cuda_ad_rgb')

# Load an arbitrary scene
scene = mi.load_dict(mi.cornell_box())
params = mi.traverse(scene)

# Make one parameter differentiable
key = 'red.reflectance.value'
dr.enable_grad(params[key])
params.update()

# Render
img = mi.render(scene, spp=1, params=params)

# Does the output have gradient tracking ? Yes.
print(f"{dr.grad_enabled(img)=}")

# Denoise
denoiser = mi.OptixDenoiser(input_size=img.shape[:2])
denoised = denoiser(img)

# Does the denoised output still have gradient tracking ? No. The OptixDenoiser is not differentiable.
print(f"{dr.grad_enabled(denoised)=}")

# Copy AD graph from the original render over to the denoised output
denoised = dr.replace_grad(denoised, img)

# Does the denoised output now have gradient tracking ? Yes.
print(f"{dr.grad_enabled(denoised)=}")

[saedrna]

Thanks. I have checked the code for nvdiffrecmc again and found that it just used a bilateral denoiser, with normal and depth from the G-buffer along with the shaded color. Although the denoiser code is in a folder named optixutils, it does not actually use OptiX AI denoiser.