Implemented mesh texturing instead of vertex colors

[iffyloop]

First of all, thank you to Tripo AI and Stability AI for open-sourcing TripoSR! I was quite surprised by the quality of the TripoSR-generated models, and even more by the fact that I can use TripoSR on my old 2016 laptop without a CUDA GPU.

As others have asked, I've run into the problem of wanting to texture meshes generated with TripoSR instead of relying on vertex colors, thus I recently implemented a texture baking system available here: https://github.com/iffyloop/TripoSR-Bake This system not only bakes the NeRF color data to texture instead of vertex colors, but the process has been split into two parts, allowing users to edit and refine the generated mesh before baking the texture to their customized mesh, since editing geometry after baking tends to ruin the UV mapping.

(I'm opening this issue since I've seen various questions on this issue tracker and elsewhere online about texturing TripoSR-generated models, but please close if this is not the appropriate place to post or if it is unhelpful self-promotion. Would also like to get any suggestions from TripoSR developers/users on how this process could be improved).

[rawwerks]

+100 for this issue!

i think i know how to solve it, but not sure i have enough time today to do a PR: https://huggingface.co/spaces/stabilityai/TripoSR/discussions/8

tl;dr - the vertex colors just need to get converted to a texture, which looks easy to do in trimesh (although i am not familiar with the package).

https://github.com/VAST-AI-Research/TripoSR/blob/00319be2f08ddd06a43edf05fbbd46b5ea1e9228/tsr/system.py#L200C17-L200C51

https://trimesh.org/trimesh.visual.color.html#trimesh.visual.color.ColorVisuals.to_texture

[iffyloop]

@rawwerks in my repo linked above, it's a little more complex since we want to allow artists to edit the mesh before baking the texture (since sometimes TripoSR outputs will need to be smoothed or modified by hand). It also allows artists to use a much higher resolution texture without needing to increase the vertex count, so you can simplify your mesh to a low number of vertices before baking the texture. If all you need to do is get the same functionality as vertex colors but using a texture, then yes, the trimesh method linked above will probably work, and it would be cool to have that feature available in this original repo.

[rawwerks]

@iffyloop - yes I'm a huge fan of the approach you've taken! I want to dive deeper, it seems very powerful.

And I'm also hoping that the TripoSR team can get to proper mesh textures with 1-2 extra lines of code.

[pookiefoof]

@iffyloop Sweet! Is it possible for you to create a PR for the texture baking part?

[iffyloop]

Sure, I'll try to have a PR ready sometime next week. Shouldn't be too difficult, just not sure when I'll have the time...

[iffyloop]

@pookiefoof Sorry it took me such a long time to get around to making the PR! It is now ready: #94 @rawwerks I tried using mesh.visual.to_texture(), but the output ended up being garbage, see the vertex-color-to-texture branch of my fork if you want to try it for yourself. I may have misunderstood how it was supposed to be used, but I think the issue is caused by only generating one pixel per vertex color, thus interpolating between the various UVs produces incorrect results when the mesh is rendered.

[DiamondGlassDrill]

@iffyloop super work thanks for all of it!!! I do have an issue i brought back the two separated files png and the "naked" obj from the #94 PR.

1. Normal Object Generator
2. The merged two files result

I double checked the merge it should work but it is not correct. Any glue what it could be?

` if is_bake_texture_enabled: self.timer.start(f"Baking Texture for Image {base_name}") bake_output = bake_texture(meshes[0], self.model, scene_codes[0], self.config["texture_resolution"], self.config["device"]) self.timer.end(f"Baking Texture for Image {base_name}") texture_path = os.path.join(output_subdir, f"{base_name}_texture.png")

                # Convert the baked texture colors to a PIL Image
                texture_image = Image.fromarray((bake_output["colors"] * 255.0).astype(np.uint8)).transpose(Image.FLIP_TOP_BOTTOM)

                self.timer.start(f"Exporting Mesh and Texture for Image {base_name}")
                xatlas.export(mesh_path, meshes[0].vertices[bake_output["vmapping"]], bake_output["indices"], bake_output["uvs"], meshes[0].vertex_normals[bake_output["vmapping"]])
                Image.fromarray((bake_output["colors"] * 255.0).astype(np.uint8)).transpose(Image.FLIP_TOP_BOTTOM).save(texture_path)
                self.timer.end(f"Exporting Mesh and Texture for Image {base_name}")

                if combine_texture:
                    # Assign the UV coordinates and material to the mesh
                    mesh = meshes[0]
                    mesh.visual = trimesh.visual.TextureVisuals(uv=bake_output["uvs"], image=texture_image)

                    # Define the output path for the combined OBJ
                    output_path = os.path.join(output_subdir, f"{base_name}_mesh.{mesh_format}")

                    self.timer.start(f"Exporting Combined Mesh and Texture for Image {base_name}")
                    mesh.export(output_path)
                    self.timer.end(f"Exporting Combined Mesh and Texture for Image {base_name}")

                logging.info(f"Mesh and Texture for Image {base_name} saved to {mesh_path} and {texture_path}")`

[rawwerks]

to me it looks like it just isn't wrapped correctly (wrong UV coordinates for the texture).

my reasoning is that i can see the grey stripes in the 2nd image, which i think should be the grey of the frame & wheel.

[iffyloop]

@DiamondGlassDrill I don't remember those PNG/OBJ files being part of the PR, can you send me the PNG which you used to generate that model?

[pookiefoof]

@DiamondGlassDrill I think to use trimesh to "re-save" the mesh, you might need to "re-index" the vertices accordingly. Maybe something like verts = meshes[0].vertices[bake_output["vmapping"]]

[DiamondGlassDrill]

Sure

[DiamondGlassDrill]

Did u find time to get on the topic? Thanks in advance for the support.

[iffyloop]

@DiamondGlassDrill I haven't had time to look at this yet but may in the next few weeks. I'm going to revisit some of this texturing stuff anyways for another project of mine; when that happens I'll try to remember to update you here also. Thank you for attaching the test image, that helps a lot!

[DiamondGlassDrill]

@iffyloop hope you are doing well!

Did you find time to look into the bike problem? Thanks into advance.

Have a nice weekend!

[pookiefoof]

@iffyloop hope you are doing well!

Did you find time to look into the bike problem? Thanks into advance.

Have a nice weekend!

Hi @DiamondGlassDrill, simply change the line mesh = meshes[0] in your code to:

mesh = trimesh.Trimesh(
    vertices=meshes[0].vertices[bake_output["vmapping"]],
    faces=bake_output["indices"],
    vertex_normals=meshes[0].vertex_normals[bake_output["vmapping"]],
    process=False
)

will do.

bicycle.zip

[DiamondGlassDrill]

@pookiefoof this worked ;) thanks :) amazing.

[DiamondGlassDrill]

Even so I do not see any resolution difference in changing resolution @iffyloop, yes there is a blowup effect from 1k to 2k 4k etc so the image size grows but not the resolution of the texture unfortunately. Will try to make some image examples to show. Any hint already why that might be ?

[iffyloop]

@DiamondGlassDrill I think that's because TripoSR is limited in how much detail it can produce, and the texture is generated simply by sampling from TripoSR's internal representation of the mesh.

Also thank you @pookiefoof for troubleshooting the previous issue!

[rawwerks]

i can't quite tell from this thread - does triposr now output gltf with mesh textures instead of vertex colors?

[rawwerks]

(in other words, does https://github.com/VAST-AI-Research/TripoSR/commit/d26e33181947bbbc4c6fc0f5734e1ec6c080956e close this issue?)

[pookiefoof]

i can't quite tell from this thread - does triposr now output gltf with mesh textures instead of vertex colors?

Yes! With @iffyloop's PR, TripoSR now can produce both mesh textures and vertex colors. So I'm closing this issue. Thanks.