Rendering time increases with consecutive renders using custom shader

osylum commented 1 year ago

Summary

Rendering time increases with consecutive renders using custom shader.

System configuration

System information:

OS: Windows-10 CPU: Intel64 Family 6 Model 165 Stepping 5, GenuineIntel GPU: NVIDIA RTX A4000 Python: 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] NVidia driver: 517.40 CUDA: 10.0.130 LLVM: 15.-1.-1

Dr.Jit: 0.4.0 Mitsuba: 3.2.0 Is custom build? False Compiled with: MSVC 19.34.31937.0 Variants: scalar_rgb scalar_spectral cuda_ad_rgb llvm_ad_rgb

Description

I have a very simple scene with a sphere, a point light and a bsdf material (custom or mitsuba one). I loop over light directions and create a new point light emitter, sphere with bsdf (in the original code I have also a loop over bsdf materials) and scene at each iteration. When using a mitsuba bsdf (e.g. diffuse), the render time remain pretty much constant. But when I use a custom bsdf (dummy diffuse in python), the render time keeps increasing. Could please help me to understand and improve the behavior?

I have also side questions in terms of rendering time:

in what order I should perform nested loops (in terms of lighting position, bsdf material, camera position) to be the most efficient? In particular with cuda_ad_rgb variant that probably sends data to gpu.
is it faster to create a scene up front and only change some properties according to the loop variables? If so, I tried to do that for the sphere object, but could not update bsdf properties, as they were not in the traverse list.

Side notes:

I think there is a small typo mistake (the lighting) in the doc (BSDFs): "In fact, the right solution in this case would be to switch to a different the lighting setup that causes more illumination to be received by the desk and then reduce the material’s reflectance"
diffuse.cpp multiplies the brdf by cos_theta_o. It is probably right, but it is not clear to me why it is not multiplied by cos_theta_i instead (<normal,light>). I still have to learn more about importance sampling, so maybe there is something I don't understand when you talk about cosine foreshortening factor - that I understand to be the shadowing term in the render equation.

Thank you

Steps to reproduce

Here is a minimal example. You change the bsdf_type in the main function. Behavior is similar when using llvm_ad_rgb versus cuda_ad_rgb variants.

import os,sys
import gc
import shutil

from timeit import default_timer as timer

import mitsuba as mi
print('available mitsuba variants:', mi.variants())
#mi.set_variant("scalar_rgb")
mi.set_variant('llvm_ad_rgb') # could not find LLVM-C.dll
#mi.set_variant('cuda_ad_rgb')
import drjit as dr

os.environ["OPENCV_IO_ENABLE_OPENEXR"] = "True"
import cv2 as cv

import imathnumpy
import numpy as np
from scipy import stats
from scipy.spatial.transform import Rotation as Rot

import imageio

class BSDF_Diffuse(mi.BSDF):
    """
    Dummy Diffuse BSDF
    """

    def __init__(self, props):
        mi.BSDF.__init__(self, props)

        self.rho_d = mi.Vector3f(props['rho_d'])

        self.m_flags = mi.BSDFFlags.DiffuseReflection | mi.BSDFFlags.FrontSide

    def sample(self, ctx, si, sample1, sample2, active):

        cos_theta_i = mi.Frame3f.cos_theta(si.wi)

        active &= cos_theta_i > 0

        bs = mi.BSDFSample3f()

        if not ctx.is_enabled(mi.BSDFFlags.DiffuseReflection):
            return (bs, 0.)

        bs.wo  = mi.warp.square_to_cosine_hemisphere(sample2)
        bs.pdf = mi.warp.square_to_cosine_hemisphere_pdf(bs.wo)
        bs.eta = 1.0
        bs.sampled_type = mi.UInt32(+mi.BSDFFlags.DiffuseReflection)
        bs.sampled_component = 0

        value = self.rho_d

        return ( bs, dr.select(active & (bs.pdf > 0.0), value, mi.Vector3f(0)) )

    def eval(self, ctx, si, wo, active):

        cos_theta_i = mi.Frame3f.cos_theta(si.wi)
        cos_theta_o = mi.Frame3f.cos_theta(wo)

        active &= (cos_theta_i > 0.0) & (cos_theta_o > 0.0)

        f_d = (self.rho_d / dr.pi) * cos_theta_o

        return dr.select(active, f_d, mi.Vector3f(0))

    def pdf(self, ctx, si, wo, active):

        if not ctx.is_enabled(mi.BSDFFlags.DiffuseReflection):
            return 0.

        cos_theta_i = mi.Frame3f.cos_theta(si.wi)
        cos_theta_o = mi.Frame3f.cos_theta(wo)

        active = (cos_theta_i > 0.0) & (cos_theta_o > 0.0)

        pdf = mi.warp.square_to_cosine_hemisphere_pdf(wo)
        return dr.select(active, pdf, 0.)

    def eval_pdf(self, ctx, si, wo, active):

        if not ctx.is_enabled(mi.BSDFFlags.DiffuseReflection):
            return (0. ,0.)

        cos_theta_i = mi.Frame3f.cos_theta(si.wi)
        cos_theta_o = mi.Frame3f.cos_theta(wo)

        active &= (cos_theta_i > 0.0) & (cos_theta_o > 0.0)

        pdf = mi.warp.square_to_cosine_hemisphere_pdf(wo)
        f_d = (self.rho_d / dr.pi) * cos_theta_o

        return (dr.select(active, f_d, mi.Vector3f(0.)), dr.select(active, pdf, 0.))

    def to_string(self):
        return ('Diffuse[\n'
                '    rho_d=%s,\n'
                ']' % (self.rho_d))

def get_lights_positions(light_dist, angle_start=0., angle_end=np.pi, numsubdivisions=90):
    """
    Get list of of lights positions

    camera is along y
    up is z
    light/envmap rotates around y from (0,-light_pos,0) to (0,light_pos,0)

    Parameters
    ---------------
    light_dist         : distance of light to center
    angle_start        : starting angle
    angle_end          : final angle
    numsubdivisions    : number of different angular subdivisions

    """

    rotation_axis = np.array([0., 0., 1.])
    light_position_start = np.array([0., -light_dist, 0.])
    angles = np.linspace(angle_start, angle_end, numsubdivisions)
    print('angles', angles)
    W = rotation_axis[np.newaxis, :] * angles[:, np.newaxis]
    R = R = Rot.from_rotvec(W)
    lights_positions = R.apply(light_position_start)
    print('lights_positions', lights_positions)

    return lights_positions

def render_sphere(bsdf_type):
    # define options
    # --------------------------------------------------------------------------

    emitter_type = 'point_light'
    envmap_dir = '../../../RND_Relighting_Data/LatLongs'
    envmap_name = 'studio.exr'
    lights_numsubdivisions = 90
    emitter_samples = 64

    dataset_merl = os.path.join("S:/mitsuba_datasets/merl", f'{emitter_type}')

    geo_type = 'sphere'

    if bsdf_type == 'diffuse_custom':
        mi.register_bsdf("diffuse_custom", lambda props: BSDF_Diffuse(props))

    rendered_aov = False  # keep track in order to only render once
    save = False

    # define camera
    # --------------------------------------------------------------------------
    camera_origin = [0., -3., 0.]
    camera_target = [0., 0., 0.]

    camera_to_world = mi.ScalarTransform4f.look_at(
        origin=camera_origin,
        target=camera_target,
        up=[0, 0, 1]
    )
    camera_fov = 38.8839

    render_resolution = [864, 486]

    sensor = mi.load_dict({
        "type": "perspective",
        "fov_axis": "x",
        "fov": camera_fov,
        "to_world": camera_to_world,
        "film": {
            "type": "hdrfilm",
            "width": render_resolution[0],
            "height": render_resolution[1],
            "pixel_format": "rgba",
            "rfilter": {"type": "lanczos"},
        }
    })

    # define lights positions
    # --------------------------------------------------------------------------
    light_dist = 3.
    lights_positions = get_lights_positions(light_dist, 0., np.pi, lights_numsubdivisions)

    # define geo
    # --------------------------------------------------------------------------
    sphere_center = [0., 0., 0.]
    sphere_radius = 0.5
    if bsdf_type == 'diffuse_custom':
        bsdf_dict = {
            'type': 'diffuse_custom',
            'rho_d': [0.5, 0.5, 0.5],
        }
    elif bsdf_type == 'mitsuba':
        bsdf_dict = {
            'type': 'plastic',
        }
    sphere = mi.load_dict({
        'type': 'sphere',
        'center': sphere_center,
        'radius': sphere_radius,
        'bsdf': mi.load_dict(bsdf_dict)
    })

    selected_materials_names = ['red-specular-plastic']

    # loop over lights positions
    for light_id, light_position in enumerate(lights_positions):

        if False:
            print(f'light {light_id}; position: {light_position}')

        if False:  # rotating the light is slow, trying to rotate the camera instead
            # but it is not faster
            camera_to_world = mi.ScalarTransform4f.look_at(
                origin=camera_origin,
                target=camera_target,
                up=[0, 0, 1]
            )
            sensor_params = mi.traverse(sensor)
            print('sensor_params.keys()', sensor_params.keys())
            sensor_params['to_world'] = camera_to_world
            sensor_params.update()

        # define emitters
        # --------------------------------------------------------------------------
        if emitter_type == 'sphere_light':
            light_intensity = 20.
            emitter = mi.load_dict({
                'type': 'sphere',
                'center': light_position,
                'radius': 0.1,
                'sphere_light': {
                    "type": "area",
                    "radiance": light_intensity,
                }
            })
        elif emitter_type == 'point_light':
            light_intensity = 2.
            emitter = mi.load_dict({
                "type": "point",
                "position": light_position,
                "intensity": light_intensity,
            })
        elif emitter_type == 'slab_light':
            slab_width = 1.
            slab_height = 3.
            slab_to_world = mi.ScalarTransform4f.look_at(
                origin=light_position,
                target=sphere_center,
                up=[0, 0, 1]
            )
            slab_to_world = slab_to_world.scale((slab_width, slab_height, 1.))
            light_intensity = 15.
            emitter = mi.load_dict({
                'type': 'rectangle',
                # 'slab_geo': slab_mesh,
                'to_world': slab_to_world,
                'flip_normals': False,
                'slab_light': {
                    "type": "area",
                    "radiance": light_intensity,
                }
            })
        elif emitter_type == 'envmap':
            envmap_filepath = os.path.join(envmap_dir, envmap_name)
            envmap_to_world = mi.ScalarTransform4f.look_at(
                origin=light_position,
                target=sphere_center,
                up=[0, 0, 1]
            )
            emitter = mi.load_dict({
                'type': 'envmap',
                'filename': envmap_filepath,
                'to_world': envmap_to_world
            })

        # loop over materials
        for material_name in selected_materials_names:

            #materialId = materials_names.index(material_name)
            #material_brdf_params = brdf_params[materialId]

            #filedir = os.path.join(dataset_merl, material_name)

            if False:
                print('material_name', material_name)
                print(f'material_brdf_params: {material_brdf_params}')

            # define bsdf
            # --------------------------------------------------------------------------
            if bsdf_type == 'diffuse_custom':
                bsdf = mi.load_dict({
                    'type': 'diffuse_custom',
                    'rho_d': [0.25012467, 0.04386179, 0.02270369],
                })
            elif bsdf_type == 'mitsuba':  # test only
                bsdf = mi.load_dict({
                    'type': 'plastic',
                })

                # update bsdf for the sphere
            if False:  # FIXME: cannot update properties
                sphere_params = mi.traverse(sphere)
                print('sphere', sphere)
                print('sphere_params', sphere_params)
                print('sphere_params.keys()', sphere_params.keys())
                sphere_params["bsdf.rho_d"] = rho_d.tolist()
                sphere_params["bsdf.rho_s"] = rho_s.tolist()
                sphere_params["bsdf.ior"] = ior
                sphere_params["bsdf.m"] = m
                sphere_params.update()
            else:
                sphere = mi.load_dict({
                    'type': 'sphere',
                    'center': sphere_center,
                    'radius': sphere_radius,
                    'bsdf': bsdf
                })

            # define scene for rendering radiance
            # --------------------------------------------------------------------------
            scene = mi.load_dict({
                'type': 'scene',
                'integrator': {
                    'type': 'direct',  # no need for indirect lighting
                    'emitter_samples': emitter_samples,
                    'hide_emitters': True
                },
                # 'point_light_emitter': point_light_emitter,
                # 'slab_emitter': slab_emitter,
                'emitter': emitter,
                # 'slab' : slab,
                'sphere': sphere,
                'sensor': sensor
            })
            # print('scene:\n', scene)

            start = timer()
            print('start render')
            img = mi.render(scene).numpy()
            print('elapsed time:', timer() - start)

            if save:
                filepath = os.path.join(filedir, "radiance_{:03d}.exr".format(light_id))
                print('saved radiance to file', filepath)
                imageio.imwrite(filepath, img[..., 0:3])

def main():

    bsdf_type = 'diffuse_custom' # mitsuba or diffuse_custom
    render_sphere(bsdf_type)

if __name__ == "__main__":
    main()

njroussel commented 1 year ago

Hi @osylum

Scene deallocation is currently a bit tricky with custom plugins. I believe that the bsdf object is leaking between iteration loops. I'll skip the details here, this involves some pretty in-depth CPython/Pybind11/Mitsuba pieces.

First, let us comfirm that this is indeed the issue. Can you move the bsdf loading outside of the loop so that it is only done once?

osylum commented 1 year ago

The rendering time stops increasing when moving these lines before the loop over light directions.


# define bsdf
# --------------------------------------------------------------------------
if bsdf_type == 'diffuse_custom':
    bsdf = mi.load_dict({
        'type': 'diffuse_custom',
        'rho_d': [0.25012467, 0.04386179, 0.02270369],
    })
elif bsdf_type == 'mitsuba':  # test only
    bsdf = mi.load_dict({
        'type': 'plastic',
    })

njroussel commented 1 year ago

Is this good enough for you? Or is there some specific reason you need to define your plugins in the inner loops?

osylum commented 1 year ago

When looping over materials outside the loop over light directions, I have an increase each time I change material. It is less strong doing the loop in this manner, but when I have many materials it still result in a significant increase. I tried also to have a loop around the render_sphere() instead, to see if there would be some initialization at each call, but it did not change the result. Though, everytime I run the .py file it the rendering time starts from a fresh state. I tried reimporting mitsuba with importlib.reload and setting the variant at each call to render_sphere(), but it did not change the behavior as well. Is there a command to reset the state of mitsuba, as it would be at the start of running the .py file?

njroussel commented 1 year ago

You could try using dr.registry_clear(). This essentially flushes the just-in-time compiler that is used by mitsuba. Once you've called this function, you can no longer use any existing variables :warning:.

osylum commented 1 year ago

I tried using dr.registry_clear() inside the materials loop as follows:


dr.registry_clear() # can no longer used any existing variable
mi.set_variant('llvm_ad_rgb')  # could not find LLVM-C.dll
# mi.set_variant('cuda_ad_rgb')
sensor = create_sensor()
object = create_object('sphere', bsdf_type, sphere_center, sphere_radius)

But at the second iteration, the program stopped with error at scene creation:

Process finished with exit code -1073740791 (0xC0000409)

The scene is:


scene = mi.load_dict({
    'type': 'scene',
    'integrator': {
        'type': 'direct', # no need for indirect lighting
        'emitter_samples': emitter_samples,
        'hide_emitters': True
    },
    'emitter':emitter,
    'object': object,
    'sensor': sensor
})

with components:


emitter:
 PointLight[
  position = [0, -3, 0],
  intensity = UniformSpectrum[value=[2]],
  medium = none]
object:
 Sphere[
  to_world = [[0.5, 0, 0, 0],
              [0, 0.5, 0, 0],
              [0, 0, 0.5, 0],
              [0, 0, 0, 1]],
  center = [0, 0, 0],
  radius = 0.5,
  surface_area = [3.14159],
  bsdf = CookTorrance[
      rho_d=[[0.03126385062932968, 0.01777799427509308, 0.014777406118810177]],
      rho_s=[[0.15480387210845947, 0.14453330636024475, 0.15086358785629272]],
      eta=1.25005616,
      roughness=0.12849635,
  ]
]
sensor:
 PerspectiveCamera[
  x_fov = [38.8839],
  near_clip = 0.01,
  far_clip = 10000,
  film = HDRFilm[
    size = [864, 486],
    crop_size = [864, 486],
    crop_offset = [0, 0],
    sample_border = 0,
    filter = LanczosSincFilter[lobes=3.000000],
    file_format = OpenEXR,
    pixel_format = rgba,
    component_format = float16,
  ],
  sampler = IndependentSampler[
    base_seed = 0
    sample_count = 4
    samples_per_wavefront = 1
    wavefront_size = 0
  ],
  resolution = [864, 486],
  shutter_open = 0,
  shutter_open_time = 0,
  to_world = [[-1, 0, 0, 0],
              [0, -0, 1, -3],
              [0, 1, 0, 0],
              [0, 0, 0, 1]]
]

Setting or not the variant does not change the behavior. Question: Is there something I might be doing wrong?

As mentioned previously, I had tried to update the object bsdf, in place of making a new call to load_dict. I used:


object_params = mi.traverse(object)

but the properties that I see are only:


object_params SceneParameters[
  ---------------------------------------------------------------
  Name        Flags    Type  Parent
  ---------------------------------------------------------------
  to_world    ∂, D     Transform4f Sphere

Question: Is it a good approach? If so, how can I set the bsdf with this approach?

njroussel commented 1 year ago

Yes this all seems reasonable. The BSDF type cannot be changed by using traverse() you need to, as you already are doing, create a new BSDF and insert it into a new scene.

Another suggestion would be to rewrite your code such that render_sphere takes a single BSDF and Emitter type, and do all the necessary mitsuba imports in that function. With this you can launch a new process that executes this function for every configuration you want to generate.

colinzhenli commented 3 weeks ago

Hi @osylum @njroussel , I am still confused with this cosine foreshortening part: diffuse.cpp multiplies the brdf by cos_theta_o. It is probably right, but it is not clear to me why it is not multiplied by cos_theta_i instead (<normal,light>). I still have to learn more about importance sampling, so maybe there is something I don't understand when you talk about cosine foreshortening factor - that I understand to be the shadowing term in the render equation. Did you understand this?

merlinND commented 2 weeks ago

Hello @colinzhenli,

Your question is unrelated to this issue, please open a new Discussion with your question if needed.

mitsuba-renderer / mitsuba3