feature request: api support for loading textures from obj/mtl file

[RussTedrake]

i asked (via email)

We're making pretty heavy use of meshcat in class again this term. I'm working on getting it into drake proper so the in-tree examples can use it, too: https://github.com/RobotLocomotion/drake/pull/9833

But the current implementation that we have uses MeshLambertMaterial for all of the objects -- including the ones loaded from obj files. It looks like Meshcat does support texture maps. But if I leave the materials field = None for that call to SetObject, then I still don't see e.g. the kuka's textures.

Is there an easy fix that is right under my nose?

and the response was

The way textures are handled in meshcat is currently pretty manual. You need to set the map field to an ImageTexture, and that texture in turn needs its image field set to a PngImage. This is much easier to explain with an example: https://github.com/rdeits/meshcat-python/blob/8b24ac3a72ea56fbb64fca5fcdb1782daa0961bf/src/meshcat/tests/test_drawing.py#L50-L57

Currently, I don't try to do something clever like parse the .mtl file to load the texture, but I suspect that would be possible, perhaps with a Mesh.from_file() constructor.

[gizatt]

This would be nice for getting textures drawn on e.g. Drake's flashy new Skydio quadcopter model, which has relatively complex material setup on its different parts. Would this be easy to do using _meshfile_object parsing that looks like has come into existence on the javascript side over the last year?

(I'm happy to take a crack at it some time, just don't have a birds-eye view if that's the right strategy.)

[rdeits]

Yeah, that's the right thing to do. I'm happy to help out with the details!

There's an example of _meshfile_object usage from javascript here: https://github.com/rdeits/meshcat/blob/master/test/meshfile_object_obj.html . Adding support in python would just mean creating a MeshFileObject type in geometry.py which lowers to the same data format as that example json data.

The hardest part of this actually getting the texture file into the right format. I'm using the native object loader in Three.js, which expects that the texture files will be at some URL that it can access. Since that's annoying, I'm instead relying on the fact that you can create a data URI which includes the actual contents of the image file as base-64 encoded data. That means that texture loading just works, as long as you're willing to jump through some hoops to encode the data. The example linked above does that. In case it's helpful, the relevant Julia code that parses an obj file for its associated .mtl and then parses that .mtl for its necessary image textures is here: https://github.com/rdeits/MeshCat.jl/blob/8fbf7286a1bd721d80d990f5ea744911fb1adc2f/src/mesh_files.jl#L77-L101