PyTorch3D renderer for normal and specular maps

[ATAboukhadra]

Hi,

Is there a way to use PyTorch3dD renderers to render the NIMBLE mesh using the diffuse maps and the normal and specular maps?

For diffuse maps, it's easy to create a Mesh object with TexturesUV object that contains the diffuse map and the uv mapping.

For normals, I found a way to create a special type of materials used for rendering that includes normal maps:

def prepare_materials(params, batch_size, shared_texture=True, device='cuda'):
    if 'normal_map' in params:
        if shared_texture:
            normal_maps = params['normal_map'][None, 0].repeat(batch_size, 1, 1, 1).to(device)
        else:
            normal_maps = params['normal_map'].to(device)

        normal_maps = torch.nn.functional.normalize(normal_maps, dim=-1)
        normal_maps = TexturesUV(maps=normal_maps,
                                faces_uvs=params['faces_uvs'].repeat(batch_size, 1, 1).type(torch.long).to(device),
                                verts_uvs=params['verts_uvs'].repeat(batch_size, 1, 1).to(device))
    else:
        normal_maps = None

    materials_prop = {
        "normal_maps": normal_maps,
    }
    return materials_prop

def create_materials(tex_img, faces_uvs, verts_uvs, bn, device='cuda'):

    normal = tex_img[..., [5, 4, 3]]
    material_params = {
        'normal_map': normal,
        'faces_uvs': faces_uvs,
        'verts_uvs': verts_uvs,
    }
    materials_properties = prepare_materials(material_params, bn, shared_texture=True, device=device)
    materials = PBRMaterials(
        device=device,
        shininess=0.0,
        **materials_properties,
    )

    return materials

The purpose is to calculate render loss with RGB images. Is there a cleaner way of including the normal and specular maps in pytorch3d rendering?

Thanks.

[reyuwei]

Hi, I am not familiar with pytorch3d. I used https://github.com/mmatl/pyrender.