PBR material not working on meshes

[chisarie]

System:

• OS version: Ubuntu 20.04
• Python version (if applicable): Python 3.10
• SAPIEN version: 2.2.2
• Environment: Desktop

Describe the bug I am trying to add custom textures to objects in sapien. It works for box primitives, but it doesn't work for a general mesh.

To Reproduce Steps to reproduce the behavior (use pastebin for code):

1. Get any .obj mesh file (without .mtl)
2. Get texture files, e.g. from https://ambientcg.com/get?file=Bricks082A_2K-JPG.zip
3. Run script to reproduce: https://pastebin.com/sintqu9v

Expected behavior I expect to see the bricks texture on both objects, but only the simple box gets textured. The same happens with and without raytracing.

Screenshots

[fbxiang]

Can you check the obj file has UV coordinates?

[chisarie]

Ah good point, the obj does not have them. I could use open3d's function compute_uvatlas to compute them. I just tried to load the mesh in open3d, compute the uvatlas, and then save the mesh again to a new file -> works very well!

But it would be also nice to do it on the fly, without saving a new .obj. I am trying to use sapien's RenderMesh class, but I have the issue that open3d returns face_uv instead of vertex_uv. This is my current code:

# Get UV Atlas (only available for tensor mesh)
o3d_mesh = o3d.t.io.read_triangle_mesh(str(mesh_path))
o3d_mesh.compute_uvatlas()
vertices = o3d.core.Tensor.numpy(o3d_mesh.vertex.positions)  # shape: (200, 3)
triangles = o3d.core.Tensor.numpy(o3d_mesh.triangle.indices)  # shape: (396, 3)
triangle_uvs = o3d.core.Tensor.numpy(o3d_mesh.triangle.texture_uvs)  # shape: (396, 3, 2)
vertex_uvs = ??? # shape: (200, 2)

# Render Mesh
render_mesh = renderer.create_mesh(vertices, triangles)
render_mesh.set_uvs(vertex_uvs)
actor_builder.add_visual_from_mesh(render_mesh, material=material)

I asked in their repo if it is possible to return vertex_uvs (see https://github.com/isl-org/Open3D/issues/6465). Do you know whether sapien could use face_uvs instead of vertex uvs?

[chisarie]

I got a response from the open3d repo, the missing snippet is:

_, ids = np.unique(triangles, return_index=True)
vertex_uvs = triangle_uvs.reshape(-1, 2)[ids]  # shape: (200, 2)

Now it works, thank you for everything :)