About Relighting Results

[JiuTongBro]

Hi, thanks for your meaningful work firstly.

Recently I am trying to reimplement your relighting results of NeRFactor Synthetic dataset in paper. I followed the same albedo-scale processing in https://github.com/NVlabs/nvdiffrecmc/issues/20#issue-1555093236, which worked for the NVDIFFRECMC model. But still I ran into some troubles in the relighting processing.

As shown in the above figure, I acquired almost the same reconstruction result in your paper, even the noise parts are very similar. However, in relighting results there are many unexpected dark areas. Do you have any idea about what leads to it? I have tried both the Blender script/setting provided in NVDIFFRECMC, and the original one you provided in https://github.com/NVlabs/nvdiffrec/issues/21#issue-1210905272, but none of them works.

I suppose is it possible that they are caused by inversed mesh surface? Is there anything else we need to do, like some normal correction, before we put the generated mesh back to the Blender? Thanks.

[JiuTongBro]

Update: It seems the dark areas are caused by lacking of alpha_transparency, that part of codes shall be kept in NVDIFFREC's relighinting. However, the corrected result is less 'reflected' than in paper.

[mohamed-ebbed]

I guess the problem may be in this part. If you are loading a .mtl file, this function is called from render/material.py

def load_mtl(fn, clear_ks=True):
    import re
    mtl_path = os.path.dirname(fn)

    # Read file
    with open(fn, 'r') as f:
        lines = f.readlines()

    # Parse materials
    materials = []
    for line in lines:
        split_line = re.split(' +|\t+|\n+', line.strip())
        prefix = split_line[0].lower()
        data = split_line[1:]
        if 'newmtl' in prefix:
            material = Material({'name' : data[0]})
            materials += [material]
        elif materials:
            if 'bsdf' in prefix or 'map_kd' in prefix or 'map_ks' in prefix or 'bump' in prefix:
                material[prefix] = data[0]
            else:
                material[prefix] = torch.tensor(tuple(float(d) for d in data), dtype=torch.float32, device='cuda')

    # Convert everything to textures. Our code expects 'kd' and 'ks' to be texture maps. So replace constants with 1x1 maps
    for mat in materials:
        if not 'bsdf' in mat:
            mat['bsdf'] = 'pbr'

        if 'map_kd' in mat:
            mat['kd'] = texture.load_texture2D(os.path.join(mtl_path, mat['map_kd']))
        else:
            mat['kd'] = texture.Texture2D(mat['kd'])

        if 'map_ks' in mat:
            mat['ks'] = texture.load_texture2D(os.path.join(mtl_path, mat['map_ks']), channels=3)
        else:
            mat['ks'] = texture.Texture2D(mat['ks'])

        if 'bump' in mat:
            mat['normal'] = texture.load_texture2D(os.path.join(mtl_path, mat['bump']), lambda_fn=lambda x: x * 2 - 1, channels=3)

        # Convert Kd from sRGB to linear RGB
        mat['kd'] = texture.srgb_to_rgb(mat['kd'])

        if clear_ks:
            # Override ORM occlusion (red) channel by zeros. We hijack this channel
            for mip in mat['ks'].getMips():
                mip[..., 0] = 0.0 

    return materials

If you look closely at this part, you will see that it is zeroing out the first channel in Ks. You can set clear_ks to False and try again.

  if clear_ks:
      # Override ORM occlusion (red) channel by zeros. We hijack this channel
      for mip in mat['ks'].getMips():
          mip[..., 0] = 0.0 

[changwoonchoi]

I was suffering from the same issue. Thank you Mohamed!