corresponding to the envphong example

[lith0613]

This demo shows how to optimize the cube map to a target cube map and using shading model of mirror reflection plus a Phong BRDF. ··· def render_refl(ldir, cpos, mvp):

Transform and rasterize.

        viewvec = pos[..., :3] - cpos[np.newaxis, np.newaxis, :] # View vectors at vertices.
        reflvec = viewvec - 2.0 * normals[np.newaxis, ...] * torch.sum(normals[np.newaxis, ...] * viewvec, -1, keepdim=True) # Reflection vectors at vertices.
        reflvec = reflvec / torch.sum(reflvec**2, -1, keepdim=True)**0.5 # Normalize.

        pos_clip = torch.matmul(pos, mvp.t())[np.newaxis, ...]
        rast_out, rast_out_db = dr.rasterize(glctx, pos_clip, pos_idx, [res, res])
        refl, refld = dr.interpolate(reflvec, rast_out, pos_idx, rast_db=rast_out_db, diff_attrs='all') # Interpolated reflection vectors.

        # Phong light.
        refl = refl / (torch.sum(refl**2, -1, keepdim=True) + 1e-8)**0.5  # Normalize.
        ldotr = torch.sum(-ldir * refl, -1, keepdim=True) # L dot R.

        # Return
        return refl, refld, ldotr, (rast_out[..., -1:] == 0)

    # Render the reflections.
    refl, refld, ldotr, mask = render_refl(lightdir, r_campos, r_mvp)

    # Reference color. No need for AA because we are not learning geometry.
    color = dr.texture(env[np.newaxis, ...], refl, uv_da=refld, filter_mode='linear-mipmap-linear', boundary_mode='cube')
    color = color + phong_rgb_t * torch.max(zero_tensor, ldotr) ** phong_exp # Phong.

···

how can I implement the real reflection model and how to replace the cube map with 2D texture image.
Can you give some advice about this useage. Thanks!

[s-laine]

The definition of a real reflection model is a broad question. I suggest consulting a computer graphics textbook such as Real-Time Rendering by Akenine-Möller et al. for a comprehensive discussion of local illumination and material models, and how to implement them in a renderer.

To represent an environment map in a 2D image, a standard azimuth-elevation parameterization is the simplest solution, but octahedral mapping is a nice and efficient low-distortion method. In principle, any map projection can be used. However, you will have to implement the mapping yourself using PyTorch, which is why I recommend using the provided cube map parameterization unless there is a particularly good reason to avoid it. In case you meant to ask how to apply a 2D texture on a mesh, please see the included earth.py example.

[lith0613]

The definition of a real reflection model is a broad question. I suggest consulting a computer graphics textbook such as Real-Time Rendering by Akenine-Möller et al. for a comprehensive discussion of local illumination and material models, and how to implement them in a renderer.

To represent an environment map in a 2D image, a standard azimuth-elevation parameterization is the simplest solution, but octahedral mapping is a nice and efficient low-distortion method. In principle, any map projection can be used. However, you will have to implement the mapping yourself using PyTorch, which is why I recommend using the provided cube map parameterization unless there is a particularly good reason to avoid it. In case you meant to ask how to apply a 2D texture on a mesh, please see the included earth.py example.

Hi slaine, thanks for your quick reply ! Currently I want to use your powerful renderer library to realize the usage as follows: Given a constructed head mesh and its conresonding 2D texture, when I input a 2D image of the same person in different lighting enviroment, then I want to optimize the real enviroment map by using the BRDF model. I will optimize this by calculating the loss between rendered image and input image. I hope when the optimization finish, I can load the enviroment map into unreal engine. Can you give some advise about this ?

[s-laine]

We haven't experimented with optimizing environment maps except in the toy "envphong" example, but our follow-up project nvdiffmodeling incorporates many other latent variables such as normal maps, textures, and material parameters, into the optimization. Please see the associated paper for details.

So, I don't have an answer ready for you, but I would probably try to approach the optimization of an environment map in a similar fashion as optimizing the material parameters in the nvdiffmodeling project.

[lith0613]

We haven't experimented with optimizing environment maps except in the toy "envphong" example, but our follow-up project nvdiffmodeling incorporates many other latent variables such as normal maps, textures, and material parameters, into the optimization. Please see the associated paper for details.

So, I don't have an answer ready for you, but I would probably try to approach the optimization of an environment map in a similar fashion as optimizing the material parameters in the nvdiffmodeling project.

Thanks ! I will turn to nvdiffmodeling for more details.