How do I render surface normal map "in tangent space" with PyTorch3D?

[The-Nerd-AI]

Hi, thank you for this amazing project!!

I have a question on how to render surface normal map "in tangent space" with PyTorch3D.

According to the great discussion between @gkioxari and @DaoyiG in Issue #865, we could easily render surface normal map "in world space" as follows:

 def phong_normal_shading(self, meshes, fragments) -> torch.Tensor:
      faces = meshes.faces_packed()  # (F, 3)
      vertex_normals = meshes.verts_normals_packed()  # (V, 3)
      faces_normals = vertex_normals[faces]
      ones = torch.ones_like(fragments.bary_coords)
      pixel_normals = interpolate_face_attributes(
          fragments.pix_to_face, ones, faces_normals
      )
      return pixel_normals

Also, we could easily render surface normal map "in view space" or "in ndc space" by transforming mesh vertices via transform():

https://github.com/facebookresearch/pytorch3d/blob/6f2212da46f3ad1a596b3e1017be2d16eaaf95f9/pytorch3d/renderer/mesh/rasterizer.py#L168

But, how could I render surface normal map "in tangent space" with PyTorch3D like this (blue is front, red is left, green is top)?

Thank you in advance!

[bottler]

What is meant by "in tangent space". In tangent space, the normal is always straight up (ahead..) , so there'd be nothing to see, just a constant color. In world space or view space would make sense.