[visualization] Support animation playback of hydroelastic contact patches in MeshCat

[DamrongGuoy]

Is your feature request related to a problem? Please describe. It is very helpful in dynamic simulation that we can playback animation faster or slower after the simulation finishes. Currently we can playback hydroelastic force and moment, but we do not support playback animation of contact patches. It is mentioned as a TODO here.

Describe the solution you'd like Add time parameter to Meshcat::SetTriangleColorMesh(..., time) , so we can change these lines to

      meshcat_->SetTriangleColorMesh(path + "/contact_surface", item.p_WV,
                                     item.faces, colors, false, time);  // <========Add time
      meshcat_->SetTransform(path + "/contact_surface",
                             RigidTransformd(-item.centroid_W), time);  // <========Add time

Describe alternatives you've considered

Additional context Right now, the animation playback looks like this. The contact patch (white circle) of the last time step persists and move around the white cylinder in the playback, which is very confusing. It's like we have a ghost contact patch moving around.

https://user-images.githubusercontent.com/42557859/230230501-92fcb69b-2e85-4684-8bff-5246f3311c44.mp4

[DamrongGuoy]

@joemasterjohn @RussTedrake

[DamrongGuoy]

With @SeanCurtis-TRI 's help, I looked around Drake and rdeits/meshcat. This is what I found:

1. SetTriangleColorMesh() uses structs to organize data packing of meshes; however, it lacks concepts of times.
2. SetAnimation() pack the message in-place for efficiency. It has concepts of times; however, it lacks concepts of meshes.
3. rdeits/meshcat/test/animation.html shows how to animate camera and boxes. SetAnimation() follows the pattern in animation.html.
4. rdeits/meshcat/test/animation2.html shows how to animate Obj meshes.

I think we will need to combine features from the above code together.

There's still an open question of whether we could change the mesh across time. The animation2.html uses a fixed mesh.

As I learn more about meshcat, I'll have better ideas.

[DamrongGuoy]

I will update hydroelastic_basics.ipynb when we fix this issue of contact-patch playback.

[DamrongGuoy]

@dmcconachie-tri said the attached mp4 in the issue description didn't play for him. I attached the GIF file of the same content here.

Drake_issue_19142.gif

[DamrongGuoy]

@dmcconachie-tri also reported the same symptom in Animations Playbak. The contact patches drifted. They are the very thin sliver of red rectangles, which are hard to see without zooming into the GIF a lot.

[DamrongGuoy]

There's a hope #20866 could help. Here's a prototype result from playing back a static HTML record from #20774.

[DamrongGuoy]

There is also a better idea than #20866 that Jeremy suggested in https://drakedevelopers.slack.com/archives/C43KX47A9/p1706836298149919?thread_ts=1706810068.483669&cid=C43KX47A9

to have our meshcat.cc and meshcat_animation.cc internal logic automatically "transpile" the call sequence of Meshcat::SetObject and Meshcat::SetProperty commands into a visible/invisible animation sequence, rather than teaching every Meshcat-using visualization class about the trick.

[jwnimmer-tri]

Here's an outline of how I recommend to implement this...

C++ class Meshcat can operate in three modes:

• (1) Not recording
• (2) Recording quietly (set_visualizations_while_recording = false)
• (3) Recording live (set_visualizations_while_recording = true)

Any given operation on C++ class Meshcat comes in ~three~ two flavors:

• (a) time_in_recording is provided (i.e., it's a timed event)
• (b) time_in_recording is either nullopt (i.e., it could be a timed event but in this case the user has decided to omit the time) or is not even a function argument (i.e., the operation can never be a timed event).

We want to maintain the following invariants:

• (i) While watching a live session, modes (1) and (3) are indistinguishable, i.e., any sequence of operations (of any flavor a or b) will look the same on the screen as they occur.
• (ii) When recording timed events (2a,3a), in the recorded animation those operations participate at their given time (adjusted to a frame boundary).
• (iii) When recording quiet timed events (2a) those operations do not affect the live display.
• (iv) When recording untimed events (2b,3b), in the recorded animation those operations are treated as if they happened prior to the animation starting.

Now imagine the operation SetObject(path=/foo, shape=shape). I'll use the Shape overload as an example, but the same reasoning applies to the other overloads as well (e.g., point cloud, in-memory mesh, line, etc).

We want to allow the user to make timed calls to SetObject:

 SetObject(path=/foo, shape=shape_t1, time=t1)
 SetObject(path=/foo, shape=shape_t2, time=t2)
 SetObject(path=/foo, shape=shape_t3, time=t3)
 ...

However, the animation concept in threejs doesn't allow timed object addition. Instead, we must add the object untimed and use timing to change the visibility property.

Here's the trick: WHEN RECORDING and a timed SetObject is requested, add it to the scene at /path/<animation>/tN. When the user requests the three operations shown above, we transpile them to these operations instead:

 SetProperty(path=/foo/<object>, property=visible, value=false, time=time_tstart)

 SetProperty(path=/foo/<animation>/t1, property=visible, value=false)
 SetObject(path=/foo/<animation>/t1, shape=shape_t1)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=true, time=time_t1)

 SetProperty(path=/foo/<animation>/t2, property=visible, value=false)
 SetObject(path=/foo/<animation>/t2, shape=shape_t2)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=false, time=time_t2)
 SetProperty(path=/foo/<animation>/t2, property=visible, value=true, time=time_t2)

 SetProperty(path=/foo/<animation>/t3, property=visible, value=false)
 SetObject(path=/foo/<animation>/t3, shape=shape_t3)
 SetProperty(path=/foo/<animation>/t2, property=visible, value=false, time=time_t3)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=true, time=time_t3)

 SetProperty(path=/foo/<animation>/t3, property=visible, value=false, time=time_tfinal)
 SetProperty(path=/foo/<object>, property=visible, value=true, time=time_final)

Remember that any calls without a time=... are not part of the animation.

Compared to our invariants, this bends the rules a little bit: the SceneNode tree browser will be slightly different when an animation has timed setobject calls, but only in places that users do not typically unfold.

It might help to show the above sequence as untimed vs timed operations:

 # Untimed initial conditions.
 SetProperty(path=/foo/<animation>/t1, property=visible, value=false)
 SetObject(path=/foo/<animation>/t1, shape=shape_t1)
 SetProperty(path=/foo/<animation>/t2, property=visible, value=false)
 SetObject(path=/foo/<animation>/t2, shape=shape_t2)
 SetProperty(path=/foo/<animation>/t3, property=visible, value=false)
 SetObject(path=/foo/<animation>/t3, shape=shape_t3)

 # Animation.
 SetProperty(path=/foo/<object>, property=visible, value=false, time=time_tstart)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=true, time=time_t1)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=false, time=time_t2)
 SetProperty(path=/foo/<animation>/t2, property=visible, value=true, time=time_t2)
 SetProperty(path=/foo/<animation>/t2, property=visible, value=false, time=time_t3)
 SetProperty(path=/foo/<animation>/t1, property=visible, value=true, time=time_t3)
 SetProperty(path=/foo/<animation>/t3, property=visible, value=false, time=time_tfinal)
 SetProperty(path=/foo/<object>, property=visible, value=true, time=time_final)

Final note: remember that times are not floats. They are integers where the time is truncated to a specific frame number (based on the fps). We should plan to use those integers in the paths (not the time as a float) to avoid weird round-off and precision problems. We should also only do the property-setting operations in frames that actually change something; there's no need to reset the object in frames where it didn't actually change.

I think that proposal meets the invariants. Someone should check me.

[DamrongGuoy]

Thank you very much, Jeremy. I confirm this typo: the third line from below in each block should be t3 instead of t1, right? Here's the diff:

[jwnimmer-tri]

Yes, I believe you're right.

[DamrongGuoy]

@jwnimmer-tri, thank you for the idea again. To help me understand better, I have drafted the code in #20959. I'll ask some questions in Reviewable.