Closed LeonidMill closed 4 years ago
Hi,
For .png
and .jpg
, the write_bitmap
function will apply a sRGB gamma ramp to the pixel color values. Therefore when loading one of those images as a numpy array
, this sRGB gamma ramp will be baked in the image. Using it as a reference, the optimization process will converge to a set of parameters that tries to mimic the effect of this gamma ramp.
A solution would be to output an EXR reference image, which is not gamma corrected:
write_bitmap('out_ref.exr', image_ref, crop_size)
# which we can load back from disk:
my_bmp_ref = Bitmap('out_ref.exr')
# and/or convert it to a numpy array
my_np_ref = np.array(my_bmp_ref)
When working with external data (e.g. photo, reference image generated by a different program), it is important to ensure that the format of the reference data matches the output of the differential rendering process (e.g. same color space).
We could maybe expose another parameter to the write_bitmap
function to specify whether to apply the sRGB gamma ramp or not (for png
and jpg
as well).
Hi @Speierers,
thank you for your answer. Unfortunately, I'm not too familiar with color spaces and gamma ramps. But if I understand this correctly and I work with external data (i.e. a photo in .png
format) I can use something like histogram matching to get the same color space as from the rendering process? However, I guess that the initial guess as output of the diff. rendering process (at the beginning of the optimization) must be at least similar in terms of light conditions?
Is there a possibility to undo the gamma correction in a .png
/ .jpg
image (if the reference image is not available in .exr
format)?
It would be great to know in what color space your reference image lives in and apply the right conversion. Not sure whether histogram matching will be good enough. This will definitely affect the set of parameters (e.g. color values) resulting from the optimization process.
In Mitsuba 2, it is possible to undo the gamma correction in the following way:
import numpy as np
import mitsuba
mitsuba.set_variant("scalar_rgb")
from mitsuba.core import Bitmap, Struct
# Load a PNG from disk (gamma-corrected)
img_srgb = Bitmap("my_image.png")
# Undo the gamma correction
img_linear = img_srgb .convert(Bitmap.PixelFormat.RGB, Struct.Type.UInt8, srgb_gamma=False)
# Similarly you can convert from Int8 to Float values
img_float = img_srgb .convert(Bitmap.PixelFormat.RGB, Struct.Type.Float32, srgb_gamma=False)
# Get the corresponding Numpy array
img_np = np.array(img_float)
Thank you for your answer. This is what I was looking for.
Hi,
is there a possibility of using a
numpy
array as reference image (i.e. from a target photo) for the differentiable rendering instead of using aenoki.cuda_autodiff.Float32
data type?I went through the tutorial for the differentiable rendering part and if I use the
image_ref
as described in the docs everything works fine. However, if I use theout_ref.png
loaded as anumpy array
as reference image (even if normalized) I get some unexpected results for the optimization of the wall. I guess that the problem lies in the range of pixel values for the reference image? If I callimage_ref.numpy()
I get floating point values that range from0 - 18.6
, while if I load theout_ref.png
as numpy array I get intensities from0 - 1
.