Human mesh labeler with clothed humans

[gonalo99]

Hi! First of all, thank you so much for the amazing repository.

I'm fairly new to Unity, and I was trying to get the meshes for the generated clothed humans. I tried to use the Human Mesh Labeler but I always get the meshes for the naked humans.

Is it possible to adapt it to get the clothed meshes as well?

[peifeng-unity]

Hi @gonalo99, thanks for reaching out! The Human Mesh Labeler will only export humans without clothes, and we don't have built-in tools to save meshes for the generated clothed humans. It might be possible to following the Human Mesh Labeler and make or modify a custom labeler to export the meshes that you need.

[gonalo99]

Thank you for the fast reply! Would you be able to give me some general pointers on how I could achieve that? From my understanding, the clothed humans are generated in HumanGenerator, and the Human Mesh Labeler receives the SkinnedMesh to be exported. How could I save the clothed mesh in a Component to pass it to the labeler in a similar way?

[peifeng-unity]

Yes, your understanding is right. You may need to create a new labeler or modify the existing one and pass the clothing meshes to the labeler to export the clothing data along with the human body. When a human is generated, the clothing items are also generated with the human game object with a Skinned Mesh Renderer attached. By default, the cloth objects are child objects of the generated human objects in the Hierarchy window in Unity. The clothing mesh data can be achieved from the Skinned Mesh Renderer of the clothes in the same way as the mesh of the human body. This is the line about how the mesh of the human body is passed to the labeler, and you could add an additional variable to pass the clothing mesh to the labeler to export the data.

[gonalo99]

I was able to do it. Thank you so much for the help!

[Henistein]

@gonalo99 Hello, I'm having the same problem you had initially, do you think you could share the changes you made to be able to store the meshes with clothes? Thanks in advance.

[gonalo99]

In HumanMeshLabeler, I changed the loop in line 77 to

foreach (var human in allHumans)
{
    // Skip any human that does not have a labeling component or is not active/enabled
    var labeling = human.GetComponent<Labeling>();
    if (labeling != null && human.isActiveAndEnabled)
    {
        // Loop through all the children of the human
        for (int i = 1; i < human.transform.childCount - 1; i++) // 0 is root, 1 is shirt, 2 is pants, 3 is shoes and 4 is hair
        {
            // Get the i-th child's transform
            Transform childTransform = human.transform.GetChild(i);

            meshEntities.Add(new HumanMeshEntity(labeling.instanceId,
                Path.Combine(k_SubFolder, $"frame_{Time.frameCount}", $"{labeling.instanceId}_{i}.obj"),
                childTransform.GetComponent<SkinnedMeshRenderer>(), exportMeshTriangles, m_MeshExportTaskManager, encoding));
        }

        meshEntities.Add(new HumanMeshEntity(labeling.instanceId,
            Path.Combine(k_SubFolder, $"frame_{Time.frameCount}", $"{labeling.instanceId}.obj"),
            human.GetComponent<SkinnedMeshRenderer>(), exportMeshTriangles, m_MeshExportTaskManager,
            encoding));
    }
}

This saves the individual meshes for the clothes in the same folder where the naked human mesh is saved. Then, I combined all the individual meshes into a single one in a python script

person = trimesh.load("frame_2/1.obj")
shirt = trimesh.load("frame_2/1_1.obj")
pants = trimesh.load("frame_2/1_2.obj")
shoes = trimesh.load("frame_2/1_3.obj")

yourList = [person, shirt, pants, shoes]

vertice_list = [mesh.vertices for mesh in yourList]
faces_list = [mesh.faces for mesh in yourList]
faces_offset = np.cumsum([v.shape[0] for v in vertice_list])
faces_offset = np.insert(faces_offset, 0, 0)[:-1]

vertices = np.vstack(vertice_list)
faces = np.vstack([face + offset for face, offset in zip(faces_list, faces_offset)])

merged__meshes = trimesh.Trimesh(vertices, faces)
merged__meshes.export("merged.obj")

[Henistein]

@gonalo99 Thank you so much, it worked very well! Just out of curiosity, did you also try to extract the textures?

[gonalo99]

I did not, my use case only required the meshes