support for texture faces that differ from geometry faces

[martinResearch]

Thank you very much for this great library.

At the moment Trimesh textured mesh instances do not contain an array that specifies the texture faces independently from the geometry faces. From the comment in unmerge_faces in texture.py:

Textured meshes can come with faces referencing vertex indices (v) and an array the same shape which references vertex texture indices (vt) and sometimes even normal (vn). Vertex locations with different values of any of these can't be considered the "same" vertex, and for our simple data model we need to not combine these vertices.

For that reason, when loading with trimesh an obj file with different indices for geometry faces (indices of 3D vertices) and for texture faces (different uv vertices), 3d vertices that lie along texture seams are duplicated so that the geometry faces and texture faces can be the same single array. As a result the mesh topology loaded in trimesh is not the same as the original mesh in the obj file, and a mesh that is watertight can be split in several disjoint meshes, which is a major problem for some applications.

Does it sound feasible to modify the trimesh.visual.texture class to support a new optional attribute called texture_faces for example, or would that involve too many changes in the library ? What would be the main changes to do ?

[kaufManu]

This seems to be somehow connected to #489 .

[martinResearch]

@kaufManu indeed the problem described #489  seems to come from the same core limitation in Trimesh: there is a single faces array that contains indices of vertices. It would be great if it could support multiple faces array of sizes nb_faces x 3 that contain respectively vertex indices, normals indices, and texture uv vertices indices. That would remove the need to duplicate vertices, normals or texture vertices (which is necessary to be able to use a single faces array, at the cost of changing the mesh topology) and would lead to a structure that matches exactly  the original data from the OBJ files. Not sure how big the implication would be on the rest of the library if that was implemented. 

As an example, In my own triangulated class I have a separate array from the faces array (that contain vertices indices) called face_uv that contains the uv vertices indices. I don't have a third array for normals indices as my renderer aims at doing differentiable rasterization, while having different vertex normals depending on the adjacent surface would make things non differentiable along the edge when rendering without some edge antialiasing.

[kaufManu]

Yes, I agree, that would be nice to have. Looking through the code, I think all these face arrays are already created when reading an OBJ file, they're just not exposed later on.

[martinResearch]

It seems that several opened issues are related to this issue i.e. are due to the fact that the Trimesh data model does not support different number of vt and v (https://github.com/mikedh/trimesh/issues/1117, https://github.com/mikedh/trimesh/issues/1291) Unfortunately, splitting the mesh into disconnected parts that correspond to texture islands in the uv space like currently done is not a solution for some applications that require to keep the original connectivity (like non-rigid ICP). It seems like it might be feasible to change the TextureVisuals class to add an optional faces_uv attribute to allow having several vt corresponding to a single v. The challenge would then probably be to not introduce any breaking change. @mikedh would you potentially approve a PR that implements support for different number of v and vt if it does not introduce breaking changes? Would you have some recommandation about where to do the changes ?