Meshcat::StaticHtml duplicated meshes

[siddancha]

Problem Say I have a bimanual setup of two identical iiwas loaded using the exact same URDF files (and hence the same mesh files). When I generate an HTML file of the meshcat animation, the size of the file is roughly double the size had I used a single iiwa arm.

It seems to me that the same meshes are being duplicated in the HTML file. It would be great if meshcat could detect that the same mesh files are used across different models and only load the file once.

Code reproduction Here's a simple test file that reproduces the issue:

from pydrake.all import Meshcat, StartMeshcat, Simulator
from manipulation.station import MakeHardwareStation, LoadScenario

def generated_html_size(yaml_str: str) -> float:
    meshcat: Meshcat = StartMeshcat()

    # load scenario, make station
    scenario = LoadScenario(data=yaml_str)
    station = MakeHardwareStation(scenario, meshcat=meshcat)
    context = station.CreateDefaultContext()
    simulator = Simulator(station, context)

    # record simulation in meshcat, generate static HTML
    meshcat.StartRecording(set_visualizations_while_recording=True)
    simulator.AdvanceTo(1.0)
    meshcat.StopRecording()
    html = meshcat.StaticHtml()
    meshcat.Delete()

    return len(html.encode('utf-8')) / 1e6  # MB

if __name__ == "__main__":
    one_iiwa_html_size = generated_html_size(
    """
    directives:
        - add_model:
            name: iiwa_left
            file: package://drake/manipulation/models/iiwa_description/urdf/iiwa14_polytope_collision.urdf

        - add_weld:
            parent: world
            child: iiwa_left::base
    """
    )

    two_iiwa_html_size = generated_html_size(
    """
    directives:
        - add_model:
            name: iiwa_left
            file: package://drake/manipulation/models/iiwa_description/urdf/iiwa14_polytope_collision.urdf

        - add_weld:
            parent: world
            child: iiwa_left::base

        - add_model:
            name: iiwa_right
            file: package://drake/manipulation/models/iiwa_description/urdf/iiwa14_polytope_collision.urdf

        - add_frame:
            name: iiwa_right_origin
            X_PF:
                base_frame: world
                translation: [0, 1, 0]

        - add_weld:
            parent: iiwa_right_origin
            child: iiwa_right::base
    """
    )

    print(f"One iiwa HTML file size: {one_iiwa_html_size:.3f} MB")
    print(f"Two iiwa HTML file size: {two_iiwa_html_size:.3f} MB")

Output

One iiwa HTML file size: 9.097 MB
Two iiwa HTML file size: 16.063 MB

Ideal output Ideally, both file sizes would be approximately same, since they use the exact same set of mesh files.

[jwnimmer-tri]

If your meshes are glTF files, then this already works. I'll leave this open as a feature request for *.obj files as well (which I agree end up duplicated).

[rpoyner-tri]

Under ordinary rules I would have assigned this to @joemasterjohn as geometry:illustration owner, but since @jwnimmer-tri has been toiling nearby, I'll assign it to him for further delegation.

[jwnimmer-tri]

I'll outline the technology here to help anyone who swings by:

When a meshfile uses external assets like *.png textures or *.bin vertex data, that's already de-duplicated in the static html via our CAS cache. The iiwa meshes are acute bad in this case because they use *.obj for the geometry and do not use texture images, so as of today nothing is de-duplicated. They have no external resources which can be cached and de-duplicated.

The reason it's easy to (and already implemented) to de-duplicate the CAS entries is because they are/were already cited via URL when loading the page. The msgpack message cites a URL, and the javascript code loads it. We can point both meshfile objects to the the same texture (etc.) URLs.

To de-duplicate the *.obj data itself, the ideal way would be to put the obj file in the CAS cache and refer to it via URL. This is not easily possible, because the Meshcat _meshfile_object message always has the mesh data inline in the message. The msgpack field is named data and has the obj content directly. There is no option to pass a url in the _meshfile_object message.

When I tried adding url to _meshfile_object in https://github.com/meshcat-dev/meshcat, sometimes the meshes didn't load correctly, so I set aside that attempt at the feature. Instead, TRI is focusing on using glTF files everywhere, which are uniformly better in all aspects -- speed, look and feel, authoring tools, source control, you name it. Soon, all of the drake/manipulation/models will use glTF files for visuals.

Back to this request -- if we want de-dup *.obj files, we could try something like #19598 where the same _meshfile_object UUID is sought for both meshes, but that has challenges with properly garbage collecting the resources (in a dynamic meshcat, not static html) as the page evolves.

The other hack we could do is when emitting static html we could write some bespoke code to look for large msgpack data and see if any of them share common long strings (maybe with some hints from the scene tree). Then instead of emitting the msgpack messages as-is, we could create named JS temporaries with the common substrings. Ick.

The other work-around to consider is run gzip on the html page. I'll bet it shrinks in half.

And finally a reminder of the best work-around: convert your meshes to glTF files, with *.bin assets. Then they already will de-duplicate.

Given that this works for all formats except *.obj, and the difficulty of creating a bespoke solution just for *.obj files, my priority to implement this will be not be very high.

[siddancha]

I locally converted the iiwa_description meshes from .obj to .gltf using the obj2gltf tool ...

... and indeed, my static HTML file size goes from 21 MB down to 9.3 MB!

@jwnimmer-tri Do you think it's worth PR'ing my changed mesh and URDF files to RobotLocomotion/models and RussTedrake/manipulation respectively?

[jwnimmer-tri]

Sure, we would welcome the help, at least on the models side. (I can't speak for Russ on his manipulation repo but I think he would appreciate it.)

Note that Drake's iiwa_description also would need to change (to point to the new meshes). I can help with that part, if you like.

Note that only the visual meshes should be converted. The collision meshes should remain as *.obj for now.

[siddancha]

Sounds good! Before I start the PR in models, should I add the glTF files alongside the obj files, or replace the obj files?

[jwnimmer-tri]

Ideally we would replace. Only if the PR coordination gets too complicated would we add (and then a bit later, remove). I'd bet we can do it all in one swoop, though.

[siddancha]

@jwnimmer-tri I'm working on the PR now. A choice I need to make while converting .obj meshes to glTF/.bin is whether to apply draco compression.

For example, draco reduces IIWA's link_0 (5458 vertices, 9484 triangles) .bin file size from 184 KB to 20 KB (9x!!).

I can confirm that draco-compressed meshes work with drake/meshcat. If all else equal, I would prefer to use this. Thoughts?

[jwnimmer-tri]

Geometry endpoints like RenderEngineVtk don't support draco compression yet, so we can't use draco compression exclusively. We still need to provide the uncompressed data.

On our roadmap, we're thinking about shipping both the uncompressed mesh data and draco mesh data side by side, and geometry consumers will pick whichever one they can use. However, that plumbing isn't hooked up well enough yet.

So for the moment, we should stick with normal uncompressed meshes. Down the road, we'll add the draco meshes side-by-side, in bulk.

[siddancha]

@jwnimmer-tri Here's the PR: https://github.com/RobotLocomotion/models/pull/36

[jwnimmer-tri]

I'll leave this open as a feature request for *.obj files as well (which I agree end up duplicated).

Enh, OBJ is really not a priority anymore.