Rendering OBJ files with sophisticated materials

[SeanCurtis-TRI]

Currently, when an OBJ is consumed for rendering, one of two things happens:

1. The whole geometry is assigned a single phong-like diffuse color, or
2. A .png file is found with the same name as the .obj and a single texture is applied to the whole mesh.

This precludes the following valueable behaviors:

1. Multiple materials per obj file.
2. Arbitrary material specifications as defined in the .mtl file.

The RenderEngineVtk currently uses the vtkOBJReader to load the file. It has been suggested that using the vtkOBJImporter may produce superior results.

We need to improve the OBJ integration to include more of the material specifications. Investigate what vtkOBJImporter provides over vtkOBJReader and devise a plan for filling in any missing data.

[jamiesnape]

In general terms, from https://blog.kitware.com/introducing-gltf-2-0-support-in-vtk-and-paraview:

[..] a reader is a filter that produces a data object (like a mesh or a multi-block dataset for instance), while an importer is a class that reads data and configures its representation in a renderer.

[tehbelinda]

Some notes/things to consider:

• Our Mesh/Convex classes just take a filename for the .obj file, but we'd also need some way to specify a .mtl file and a path for all the textures. We could just assume the same name but change the file extension for the .mtl (like what the current texture impl does) and assume an identically named folder in the same directory for the textures
• The OBJ importer does not expose the PolyData that OBJ reader used to, so it's not simple to apply things like TransformFilters to it. Currently there is a TransformFilter for scale. Since they are added directly to the renderer, they can be retrieved via renderer->GetActors, but this could already have other actors so how would we find just the ones that were imported? We'd also need them for making the 3 different types of actors - Color, Depth and Label.

Excerpt from https://lorensen.github.io/VTKExamples/site/Cxx/IO/OBJImporter/ which uses data from https://github.com/lorensen/VTKExamples/tree/master/src/Testing/Data/doorman

importer->SetFileName(argv[1]);
importer->SetFileNameMTL(argv[2]);
importer->SetTexturePath(argv[3]);
importer->SetRenderWindow(renWin);
importer->Update();

[SeanCurtis-TRI]

It's worth noting, that the OBJ specification includes the usemtl some_file.mtl as contents of the OBJ. That is the canonical way for an OBJ to be associated with an mtl file.

[rpoyner-tri]

Also related? #16143

[RussTedrake]

I had about 5 project groups all run into the "RenderEngineVtk doesn't support mtl" issue in the last few days. It's documented here, but not one of those groups found that documentation (fortunately, a different student in the class DID and recommended it!). I think it has surfaced as one of the major usability issues.

Compare this to supporting only .obj files. People see this as a big limitation, but they grok and easily work around it. In the mtl case they experience loading objects that render properly with textures in meshcat, and but then silently fail to load the textures in RgbdSensor. And they don't know why.

If it's hard to close this issue properly, then should we issuing a warning and give some sort of help to the user?

[RussTedrake]

Also, the PNG trick doesn't work for all obj/mtl files. For instance this one doesn't have a PNG associated with it, only mtl Kd's. simpleBoard.zip

[SeanCurtis-TRI]

Two quick questions:

1. So, one possible short-term band-aid is to give a runtime warning. Something along the lines of, "You've specified an obj for perception. It has an mtl file. We're ignoring it. If you want a texture, apply it by playing the .png game." Yes?
2. In the case of the simple board, the "PNG trick doesn't work" means that the outcome isn't desirable. There's no technical reason preventing one from applying it and it doesn't cause errors. What it does cause is a chess board that (I assume) doesn't look like a chessboard. From this, the PNG trick should be further documented to suggest it's only appropriate when the object has texture coordinates consistent with mapping all the geometry to a single image.
   • In the case of the board, the texture coordinates are a bit....crazy. It's got some implicit scaling there and they may be disjoint.
   • The example heightens the significance of the "multiple material' case -- squares of the board are defined in geometry and they are explicitly colored with different materials.

[SeanCurtis-TRI]

Another related issue is to make sure the GL renderer maintains parity. It would be regrettable if it were to fall behind.

[RussTedrake]

@SeanCurtis-TRI -- I agree with your (1) and (2). Of course, i would much prefer to just support (more) mtl files, even if we can't support the crazy ones.

For (2), the student also asked me if we knew about any tools that would load this mtl file properly, and convert it into a png texture map.

[adamconkey]

A hearty +1 for better OBJ support and, at the very least, very explicit description of what is supported in the documentation for RgbdSensor in PyDrake. I had the misfortune of burning several days messing with my mesh in Blender and Meshlab trying to export a texture PNG that would color my mesh correctly. I saw the bit about the PNG filename needing to match the OBJ filename, but I didn't realize it only supported exactly one material -- my exports from Meshlab had 12 different materials so the RGB rendering in Drake was only coloring the first material in the list.

A trick that ended up working for me in the end, if you can get an OBJ and associated PNG texture (e.g. from Meshlab export), you can edit the MTL file to look like

# my_mesh.obj.mtl (This is MTL file associated with my_mesh.obj and texture my_mesh.png)

newmtl material_0
map_Kd my_mesh.png  

And then, importantly, edit your OBJ file to make every usemtl tag refer to usemtl material_0. If it came from Meshlab and you have multiple materials you'll see other ones like usemtl material_1, use_material_2. and so forth that should all be changed to usemtl material_0.

[kevinzakka]

FYI, I am open to make obj2mjcf simulator agnostic and would be willing to invest time to make it usable by the Drake team.

[jwnimmer-tri]

FYI

@adamconkey and @kevinzakka thank you both for the info! Over the coming weeks, Drake team will be prioritizing texture loading (or at minimum, better documentation and warnings about ignored data) and I hope to be able to ping you again for help / testing / suggestions and/or try out obj2mjcf as part of our exploration.

[adamconkey]

Regarding my previous comments about the mesh with multiple materials, now that I know how to correctly export the OBJ from Blender, I'd say the issue I describe will likely only come up in rare cases. When I do a correct texture baking to a PNG in Blender, assign the texture, and then export to OBJ, I get a MTL file with only one material and reference to one PNG.

I've actually had a hard time recreating the scenario I had before where I exported an OBJ from Meshlab and got multiple materials referring to the same PNG. In that case I had a mesh that was colored but did not have an associated PNG texture. So I did various color/texture transfer operations to get it to a PNG texture that would export with the OBJ. But trying to do it again I struggle to get an OBJ to correctly export at all, let alone with these peculiarities I saw before.

So, I think as long as the OBJ files people are using come from a proper export with a baked PNG texture, it should be fine.

[SeanCurtis-TRI]

@adamconkey You'd be doing other users a huge service if you could provide a link or a more detailed rundown of the blender steps to produce this simplified OBJ.

That said, having multiple materials, etc., is a perfectly valid OBJ/MTL and Drake needs to handle those as well. :)

[adamconkey]

@SeanCurtis-TRI I followed a couple tutorials from Ryan King on YouTube, his videos are the most comprehensive and easiest to follow ones I've found. In particular I followed these two, but it looks like he has a few others as well on this topic:

1. Texture Baking for Beginners (Blender Tutorial)
2. Bake Multiple Materials to One Texture Map (Blender Tutorial)

One gotcha I ran into is make sure the PNG texture is large enough to support the UV mapping you create. I had a simple texture so I assumed I could get away with a smaller texture image (1024x1024), but I didn't account for how detailed some of the parts were in my mesh. So it let me bake the 1k image but when I added it to the mesh many colors were in the wrong spots. The fix was to create a larger 4k image (4096x4096) and then that gave enough room for all the pieces to spread out well on the UV map and the correct colors could get assigned to the parts.

The step in the second tutorial where he deletes the procedural textures after baking the PNG image, and then assigning the PNG texture to the objects is a crucial step. If you just generate a PNG image from whatever textures you have and save the image without assigning it as the texture, when you export the OBJ it will make no mention of the PNG in the MTL file and then renderers won't pick up on it. So the high-level steps are as follows (where each step is described in Ryan's videos)

1. Add textures to all objects in your mesh (e.g. procedurally generated, manually colored, from other images, etc.). This is done in the Shading tab in Blender.
2. Create a UV map in the UV Editing tab in Blender. The Smart UV Project option typically does a nice job. That will map mesh pieces to coordinates in a PNG image. Done in UV Editing tab in Blender
3. Assign the UV map you created to the objects in your mesh. Done in right-side panel in the Object Properties tab.
4. Bake the UV map, done in right-side panel in the Rendering tab. When you select Cycles as the engine (instead of default Eevee) you'll get a Bake dropdown show up. You'll probably want to do Diffuse, and deselect Direct and Indirect so that you're not baking lighting and only baking the color texture (if you watch Ryan's video he also bakes normals and metallics, these are done separately). Baking can take a while if you have a large image and/or detailed textures.
5. Save the baked image as a PNG when you're done. Can be done from UV Editing tab in Blender. You'll want to name the PNG the same base name as your mesh so it gets picked up by Drake.
6. [IMPORTANT] Go through each object in the right-side panel, and in the Material Properties tab, delete all materials associated with the object. This part is maybe a little scary so you may want to duplicate your .blend file before going this route. In the end you'll see an untextured, uncolored mesh. But, if you go to the Shading tab, you can assign your saved PNG texture map as an Image Texture. Once you wire it up to the Base Color of the Principled BSDF in the node editor, you should see your mesh get magically colored with all of the textures you had before.
7. Export as OBJ. You may need to change some settings to get the desired output, e.g. setting the forward/up axes. Since you assigned the PNG texture in the previous step, it will get referenced in your MTL file.

Again, I'd recommend watching Ryan's video and following along there, he goes through these steps in detail. Hope that helps!

[SeanCurtis-TRI]

For anyone that ends up discovering this issue and wondering what the state of OBJ is. It's still flagging. However, there's a path available to you that wasn't available when the issue was first posted. You can swap from .obj to .gltf. The glTF can be arbitrarily complex (well, no skinning, morphing, or animation) but can include multiple discrete objects, materials, etc., making use of the full PBR specification.

Specifying a <visual> tag with a .gltf file gives you support using RenderEngineVtk and the meshcat visualizer. So, there's a good alternate route to get better looking objects. Furthermore, development in this regard is on-going.