Differentiable rendering - reference image as numpy array

[LeonidMill]

Hi,

is there a possibility of using a numpy array as reference image (i.e. from a target photo) for the differentiable rendering instead of using a enoki.cuda_autodiff.Float32 data type?

I went through the tutorial for the differentiable rendering part and if I use the image_ref as described in the docs everything works fine. However, if I use the out_ref.png loaded as a numpy array as reference image (even if normalized) I get some unexpected results for the optimization of the wall. I guess that the problem lies in the range of pixel values for the reference image? If I call image_ref.numpy() I get floating point values that range from 0 - 18.6, while if I load the out_ref.png as numpy array I get intensities from 0 - 1.

[Speierers]

Hi,

For .png and .jpg, the write_bitmap function will apply a sRGB gamma ramp to the pixel color values. Therefore when loading one of those images as a numpy array, this sRGB gamma ramp will be baked in the image. Using it as a reference, the optimization process will converge to a set of parameters that tries to mimic the effect of this gamma ramp.

A solution would be to output an EXR reference image, which is not gamma corrected:

write_bitmap('out_ref.exr', image_ref, crop_size)

# which we can load back from disk:
my_bmp_ref = Bitmap('out_ref.exr')

# and/or convert it to a numpy array
my_np_ref = np.array(my_bmp_ref)

When working with external data (e.g. photo, reference image generated by a different program), it is important to ensure that the format of the reference data matches the output of the differential rendering process (e.g. same color space).

We could maybe expose another parameter to the write_bitmap function to specify whether to apply the sRGB gamma ramp or not (for png and jpg as well).

[LeonidMill]

Hi @Speierers,

thank you for your answer. Unfortunately, I'm not too familiar with color spaces and gamma ramps. But if I understand this correctly and I work with external data (i.e. a photo in .png format) I can use something like histogram matching to get the same color space as from the rendering process? However, I guess that the initial guess as output of the diff. rendering process (at the beginning of the optimization) must be at least similar in terms of light conditions?

Is there a possibility to undo the gamma correction in a .png / .jpg image (if the reference image is not available in .exr format)?

[Speierers]

It would be great to know in what color space your reference image lives in and apply the right conversion. Not sure whether histogram matching will be good enough. This will definitely affect the set of parameters (e.g. color values) resulting from the optimization process.

In Mitsuba 2, it is possible to undo the gamma correction in the following way:

import numpy as np
import mitsuba
mitsuba.set_variant("scalar_rgb")
from mitsuba.core import Bitmap, Struct

# Load a PNG from disk (gamma-corrected)
img_srgb = Bitmap("my_image.png")

# Undo the gamma correction
img_linear = img_srgb .convert(Bitmap.PixelFormat.RGB, Struct.Type.UInt8, srgb_gamma=False) 

# Similarly you can convert from Int8 to Float values
img_float = img_srgb .convert(Bitmap.PixelFormat.RGB, Struct.Type.Float32, srgb_gamma=False) 

# Get the corresponding Numpy array
img_np = np.array(img_float)

[LeonidMill]

Thank you for your answer. This is what I was looking for.