Meshcat static html: Setting colors does not work in animation

[sadraddini]

What happened?

I set meshcat_->SetProperty(*, *, *, time_in_recording) after meshcat_->StartRecording() but I don't see those properties appear when I export the recording with StaticHtml().

Here is a minimal breaking example

#include <cstdio>
#include <fstream>
#include <iostream>

#include "drake/common/find_resource.h"
#include "drake/geometry/meshcat.h"
#include "drake/geometry/shape_specification.h"
#include "drake/math/rigid_transform.h"

namespace drake {
namespace geometry {

using math::RigidTransformd;
using math::RotationMatrixd;

int do_main() {
  auto meshcat = std::make_shared<Meshcat>();

  meshcat->SetObject("sphere", Sphere(0.25), Rgba(1.0, 0, 0, 1));
  meshcat->SetTransform("sphere", RigidTransformd());

  meshcat->StartRecording();

  // change the color of the sphere
  meshcat->SetProperty("sphere", "color", {1, 0, 0, 1}, 0.0);
  meshcat->SetProperty("sphere", "color", {0, 1, 0, 1}, 1.0);
  meshcat->SetProperty("sphere", "color", {0, 0, 1, 1}, 2.0);
  meshcat->SetProperty("sphere", "color", {0, 1, 1, 1}, 3.0);
  meshcat->SetProperty("sphere", "color", {1, 1, 0, 1}, 4.0);
  // change if the sphere is visible
  meshcat->SetProperty("sphere", "visible", true, 0.0);
  meshcat->SetProperty("sphere", "visible", false, 1.0);
  meshcat->SetProperty("sphere", "visible", true, 2.0);
  meshcat->SetProperty("sphere", "visible", false, 3.0);
  meshcat->SetProperty("sphere", "visible", true, 4.0);

  meshcat->PublishRecording();

  const std::string filepath = FindResourceOrThrow("drake/geometry/test/minimal_example.html");
  const auto html_string = meshcat->StaticHtml();
  std::ofstream out;
  out.open(filepath, std::ofstream::out);
  out << html_string;
  out.close();
  return 0;
}

}  // namespace geometry
}  // namespace drake

int main() {
  return drake::geometry::do_main();
}

What I see is that colors changes don't happen in the static html (I see the last color all along) but turning the visibility on and off does.

Version

No response

What operating system are you using?

Ubuntu 20.04, Ubuntu 22.04

What installation option are you using?

binary tar.gz download, compiled from source code using Bazel

Relevant log output

No response

[SeanCurtis-TRI]

Quick question to reconcile the issue title and issue contents. The title suggests that colors simply don't animate -- and that would be the case, whether or not it also appears in the static html. Can you confirm that they simply don't animate? (Trying to save myself 20 minutes of poking around... :) )

[sadraddini]

Without passing time_in_recording to SetProperty, the colors change in a live meshcat session.

In StaticHtml, I see the last color being fixed for the whole duration of the recording.

[jwnimmer-tri]

The question is -- if the original live session, if you replay the animation do the colors dance? I suspect the answer is "no".

In Meshcat.js handling, setting the color does something out of the ordinary (finds materials and sets their colors). The animation trajectory keyframes are probably setting object.color directly, which is not a thing. The animation code will need special logic for colors, somehow.

[jwnimmer-tri]

Per f2f -- for the record, the work-around is the usual thing for missing animation features: add copies of the object(s) with all of the colors you could possibly want, set them to invisible, and then toggle their visibility in the animation.

[jwnimmer-tri]

More generally -- for SetProperty calls, anything that uses custom handling on the JS side as part of this giant if-else chain (only quoting the first few lines) ...

https://github.com/meshcat-dev/meshcat/blob/d9a2319891af5fbd04525bb4fddce355a5bad48a/src/index.js#L606-L610

... is at risk of not performing correctly during an animation. For all of them, we should either fix them to work correctly during an animation, or some warn or fail-fast when the user attempts to use them in a way that will not perform correctly.

[SeanCurtis-TRI]

I think that function may be a red herring. When you provide a time to the property, it takes a radically different path. It goes into an animation which gets almost 100% managed by three.js.

I suspect that if you try animating the color not by setting the "color" property, but, instead, by setting the "material.color" property, it may work. Certainly, when we animate opacity in meshcat_manual_test, we specify material.opacity. Even though opacity appears in that giant if-else, animating opacity goes around that particular function.

[jwnimmer-tri]

If an animation sequence with exclusively non-null timestamps doesn't replay the same thing that the user sees in live action, that's always a bug in drake. If we allow the user to set "color" and have it change the color live, then the MeshcatAnimation code needs to follow suit and transpile the property name so that is effective during replay.

Setting a different property may work around the problem, but the bug remains.

[SeanCurtis-TRI]

I wasn't implying we didn't have a bug. Merely that the bug wasn't where we might have thought it was. Certainly, the fact that meshcat aliases some of the nested properties leads to confusion.

I believe one approach that might work is to take the animation clips that get passed to meshcat, detect the aliiases supported in the big if-else block, and promote them to their true spellings. Then the alias acceptable to SetProperty() without time would also be acceptable to the animation.

I kind of feel it'd be nice if we could kill the aliases and better document and educate the common properties by their full name.