MeshCat is a remotely-controllable 3D viewer, built on top of three.js. The MeshCat viewer runs in a browser and listens for geometry commands over WebSockets. This makes it easy to create a tree of objects and transformations by sending the appropriate commands over the websocket.
The MeshCat viewer is meant to be combined with an interface in the language of your choice. Current interfaces are:
MeshCat can be used programmatically from JS or over a WebSocket connection.
To create a new MeshCat viewer, use the Viewer
constructor:
let viewer = new MeshCat.Viewer(dom_element);
where dom_element
is the div
in which the viewer should live. The primary interface to the viewer is the handle_command
function, which maps directly to the behaviors available over the WebSocket.
Viewer.handle_command(cmd)
cmd
should be a JS object with at least the field type
. Available command types are:
set_object
delete
command first (see below).
Internally, we append a final path segment, <object>
, to the provided path before creating the object. This is done because clients may disagree about what the "intrinsic" transform of a particular geometry is (for example, is a "Box" centered on the origin, or does it have one corner at the origin?). Clients can use the matrix
field of the JSON object to store that intrinsic transform, and that matrix will be preserved by attaching it to the <object>
path. Generally, you shouldn't need to worry about this: if you set an object at the path /meshcat/foo
, then you can set the transform at /meshcat/foo
and everything will just work.
Additional fields:
path
"/"
-separated string indicating the object's path in the scene tree. An object at path "/foo/bar"
is a child of an object at path "/foo"
, so setting the transform of (or deleting) "/foo"
will also affect its children.
object
toJSON()
method of a Three.js Object3D).
Beyond the nominal format, Meshcat also offers a few extensions for convenience:
geometries
stanza, the type
field can be set to "_meshfile_geometry"
to parse using a mesh file format. In this case, the geometry must also have a field named format
set to one of "obj"
or "dae"
or "stl"
and a field named "data"
with the string contents of the file.
materials
stanza, the type
field can be set to "_text"
to use a string as the texture (i.e., a font rendered onto an image). In this case, the material must also have fields named font_size
(in pixels), font_face
(a string), and text
(the words to render into a texture).
object
stanza (i.e., the object with a uuid, not the object argument to set_object), the type
field can be set to "_meshfile_object"
to parse using a mesh file format. In this case, the geometries
and materials
and geometry: {uuid}
and material: {uuid}
are all ignored, and the object must have a field named format
set to one of "obj"
or "dae"
or "stl"
and a field named "data"
with the string contents of the file. When the format is obj, the object may also have a field named mtl_library
with the string contents of the associated mtl file.
Example (nominal format):
{ type: "set_object", path: "/meshcat/boxes/box1", object: { metadata: {version: 4.5, type: "Object"}, geometries: [ { uuid: "cef79e52-526d-4263-b595-04fa2705974e", type: "BoxGeometry", width: 1, height: 1, depth:1 } ], materials: [ { uuid: "0767ae32-eb34-450c-b65f-3ae57a1102c3", type: "MeshLambertMaterial", color: 16777215, emissive: 0, side: 2, depthFunc: 3, depthTest: true, depthWrite: true } ], object: { uuid: "00c2baef-9600-4c6b-b88d-7e82c40e004f", type: "Mesh", geometry: "cef79e52-526d-4263-b595-04fa2705974e", material: "0767ae32-eb34-450c-b65f-3ae57a1102c3" } } }Note the somewhat indirect way in which geometries and materials are specified. Each Three.js serialized object has a list of geometries and a list of materials, each with a UUID. The actual geometry and material for a given object are simply references to those existing UUIDs. This enables easy re-use of geometries between objects in Three.js, although we don't really rely on that in MeshCat. Some information about the JSON object format can be found on the Three.js wiki.
Example (_meshfile_geometry
):
{ type: "set_object", path: "/some/file/geometry", object: { metadata: { version: 4.5, type: "Object" }, geometries: [ { type: "_meshfile_geometry", uuid: "4a08da6b-bbc6-11ee-b7a2-4b79088b524d", format: "obj", data: "v -0.06470900 ..." } ], images: [ { uuid: "c448fc3a-bbc6-11ee-b7a2-4b79088b524d", url: "data:image/png;base64,iVBORw0KGgoAAA==" } ], textures: [ { uuid: "d442ea92-bbc6-11ee-b7a2-4b79088b524d", wrap: [1001, 1001], repeat: [1, 1], image: "c448fc3a-bbc6-11ee-b7a2-4b79088b524d" } ], materials: [ { uuid: "4a08da6e-bbc6-11ee-b7a2-4b79088b524d", type: "MeshLambertMaterial", color: 16777215, reflectivity: 0.5, map: "d442ea92-bbc6-11ee-b7a2-4b79088b524d" } ], object: { uuid: "4a08da6f-bbc6-11ee-b7a2-4b79088b524d", type: "Mesh", geometry: "4a08da6b-bbc6-11ee-b7a2-4b79088b524d", material: "4a08da6e-bbc6-11ee-b7a2-4b79088b524d" } } }
Example (_text
):
{ type: "set_object", path: "/meshcat/text", object: { metadata: { version: 4.5, type: "Object" }, geometries: [ { uuid: "6fe70119-bba7-11ee-b7a2-4b79088b524d", type: "PlaneGeometry", width: 8, height: 8, widthSegments: 1, heightSegments: 1 } ], textures: [ { uuid: "0c8c99a8-bba8-11ee-b7a2-4b79088b524d", type: "_text", text: "Hello, world!", font_size: 300, font_face: "sans-serif" } ], materials: [ { uuid: "6fe7011b-bba7-11ee-b7a2-4b79088b524d", type: "MeshPhongMaterial", transparent: true, map: "0c8c99a8-bba8-11ee-b7a2-4b79088b524d", } ], object: { uuid: "6fe7011c-bba7-11ee-b7a2-4b79088b524d", type: "Mesh", geometry: "6fe70119-bba7-11ee-b7a2-4b79088b524d", material: "6fe7011b-bba7-11ee-b7a2-4b79088b524d", } } }
Example (_meshfile_object
):
{ type: "set_object", path: "/meshcat/wavefront_file", object: { metadata: {version: 4.5, type: "Object"}, object: { uuid: "00c2baef-9600-4c6b-b88d-7e82c40e004f", type: "_meshfile_object", format: "obj", data: "mtllib ./cube.mtl\nusemtl material_0\nv 0.0 0.0 0.0 ...", mtl_library: "newmtl material_0\nKa 0.2 0.2 0.2\n ...", resources: {"cube.png": "data:image/png;base64,iV ..."} } } }Check
test/meshfile_object_obj.html
for the full demo.
set_transform
"/foo"
will move the objects at "/foo/box1"
and "/foo/robots/HAL9000"
.
Additional fields:
path
"/"
-separated string indicating the object's path in the scene tree. An object at path "/foo/bar"
is a child of an object at path "/foo"
, so setting the transform of (or deleting) "/foo"
will also affect its children.
matrix
Float32Array
in column-major order.
{ type: "set_transform", path: "/meshcat/boxes", matrix: new Float32Array([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0.5, 0.0, 0.5, 1]) }
delete
Additional fields:
path
"/"
-separated string indicating the object's path in the scene tree. An object at path "/foo/bar"
is a child of an object at path "/foo"
, so setting the transform of (or deleting) "/foo"
will also affect its children.
{ type: "delete", path: "/meshcat/boxes" }
set_property
Note: as we append an extra path element with the name <object>
to every item created with set_object
, if you want to modify a property of the object itself, rather than the group containing it, you should ensure that your path is of the form /meshcat/foo/<object>
Additional fields:
property
visible: bool
position: number[3]
quaternion: number[4]
scale: number[3]
color: number[4]
opacity: number
(this is the same as the 4th element of color
)
modulated_opacity: number
top_color: number[3]
(only for the Background)
bottom_color: number[3]
(only for the Background)
THREE.Object3D
object. This provides a powerful capability to customize the scene, but should be considered an advanced usage -- you're on your own to avoid any unwanted side-effects.
Properties can be *chained*. For example, for an object with a phong material (MeshPhongMaterial), we may want to tweak its shininess, making it duller. Shininess is not a property of the object itself, but the object's material. There is no *path* to that material, but the property name can include a property name chain, e.g., `material.shininess`. While setting the property, the chained properties will be evaluated in sequence, such that the final name in the chain is the property that receives the `value`.
As noted, specifying a `path` that doesn't exist creates that path. However, specifying a property that doesn't exist does *not* create that property. If a name in the property chain is missing, an error message will be printed to the console and no value will be assigned. This is not a no-op per se. If the `path` led to the implicit creation of a new folder and object, that pair will still be in place.
More subtly, if the property name chain has an interior name (e.g., the `foo` in `material.foo.color`) that exists but is not an object and does not have properties (such as if `foo` were a `Number`), then, again, an error gets written to the console and no value will be assigned.
Finally, property chains can include arrays, such as `"children[1].material.specular"`. The index will be evaluated as a property (with all of the potential consequences as outlined above). In error messages, it may be reported as `children.1` instead of `children[1]`.
value
{ type: "set_property", path: "/Cameras/default/rotated/<object>", property: "zoom", value: 2.0 }Example 2:
{ type: "set_property", path: "/Lights/DirectionalLight/<object>", property: "intensity", value: 1.0 }Example 3 (chained properties):
{ type: "set_property", path: "/Lights/SpotLight/<object>", property: "shadow.radius", value: 1.0 }
set_animation
Additional fields:
animations
path
set_property
above, you will need to append <object>
to the path to set an object's intrinsic property.
clip
AnimationClip
in JSON form. The clip in turn has the following fields:
fps
name
tracks
name
field, with a single "."
before that property name to signify that it applies to exactly the object given by the path
above.
Each track has the following fields:
name
"."
(e.g. ".position"
)
type
"vector3"
for the position
property)
keys
time
(in frames) and value
indicating the value of the animated property at that time.
options
play
repetitions
{ type: "set_animation", animations: [{ path: "/Cameras/default", clip: { fps: 30, name: "default", tracks: [{ name: ".position" type: "vector3", keys: [{ time: 0, value: [0, 1, .3] },{ time: 80, value: [0, 1, 2] }], }] } },{ path: "/meshcat/boxes", clip: { fps: 30, name: "default", tracks: [{ name: ".position" type: "vector3", keys: [{ time: 0, value: [0, 1, 0] },{ time: 80, value: [0, -1, 0] }], }] } }], options: { play: true, repetitions: 1 } }
set_target
Example:
{ "type": "set_target", value: [0., 1., 0.] }This sets the camera target to `(0, 1, 0)`
capture_image
{ "type": "capture_image", "xres": 1920, "yres": 1080 }This sets the camera target to `(0, 1, 0)`
set_render_callback
{ "type": "set_render_callback", "callback": `() => { if (this.is_perspective()) { if (this.connection.readyState == 1 /* OPEN */) { this.connection.send(msgpack.encode({ 'type': 'camera_pose', 'camera_pose': this.camera.matrixWorld.elements })); } } }` }This dispatches the camera's pose in the world to the websocket connection.
Viewer.connect(url)
msgpack.decode()
from msgpack-javascript and the resulting object will be passed directly to Viewer.handle_command()
as documented above.
Note that we do support the MsgPack extension types listed in msgpack-javascript#extension-types, with additional support for the Float32Array
type which is particularly useful for efficiently sending point data and for Uint32Array
, Uint8Array
, and Int32Array
.
Objects can have their opacity changed in the obvious way, i.e.:
{ type: "set_property", path: "/path/to/my/geometry", property: "opacity", value: 0.5 }
This would assign the opacity value 0.5 to all of the materials found rooted
at "/path/to/my/geometry"
. (That means using the path "/path"
could affect
many geometries.) However, this will overwrite whatever opacity the geometry had
inherently (i.e., from the material defined in the mesh file); a transparent
object could become more opaque.
Meshcat offers a pseudo property "modulated_opacity". Meshcat remembers an object's inherent opacity and, by setting this value, sets the rendered opacity to be the product of the inherent and modulated opacity value. The corresponding command would be:
{ type: "set_property", path: "/path/to/my/geometry", property: "modulated_opacity", value: 0.5 }
Setting "modulated_opacity"
to 1 will restore the geometry's original
opacity. This does not introduce a queryable modulated_opacity
property on
any material. This is what makes it a "pseudo" property. The same tree-based
scope of effect applies to "modulated_opacity"
as with "opacity"
(and also
the "color"
property).
Meshcat always remembers the inherent opacity value. So, if you've overwritten
the value (via setting "opacity"
or "color"
), you can restore it by setting
the "modulated_opacity"
value to 1.0.
The default MeshCat scene comes with a few objects at pre-set paths. You can replace, delete, or transform these objects just like anything else in the scene.
/Lights/DirectionalLight
/Lights/AmbientLight
/Grid
/Axes
/Cameras
/Background
The camera is just another object in the MeshCat scene, so you can move it around with set_transform
commands like any other object. You can also use set_target
to change the camera's target. Please note that replacing the camera with set_object
is not currently supported (but we expect to implement this in the future).
Controlling the camera is slightly more complicated than moving a single object because the camera actually has two important poses: the origin about which the camera orbits when you click-and-drag with the mouse, and the position of the camera itself. In addition, cameras and controls in Three.js assume a coordinate system in which the Y axis is upward. In robotics, we typically have the Z axis pointing up, and that's what's done in MeshCat. To account for this, the actual camera lives inside a few path elements:
/Cameras/default/rotated/<object>
The /rotated
path element exists to remind users that its transform has been rotated to a Y-up coordinate system for the camera inside.
There is one additional complication: the built-in orbit and pan controls (which allow the user to move the view with their mouse) use the translation of only the intrinsic transform of the camera object itself to determine the radius of the orbit. That means that, in practice, you can allow the user to orbit by setting the position
property at the path /Cameras/default/rotated/<object>
to a nonzero value like [2, 0, 0]
, or you can lock the orbit controls by setting the position
property at that path to [0, 0, 0]
. Remember that whatever translation you choose is in the rotated, Y-up coordinate system that the Three.js camera expects. We're sorry.
To move the camera's center of attention to the point [1, 2, 3]
, while still allowing the user to orbit and pan manually, we suggest setting the transform of the /Cameras/default
path to whatever center point you want and setting the position
property of /Cameras/default/rotated/<object>
to [2, 0, 0]
. That means sending two commands:
{
type: "set_transform",
path: "/Cameras/default",
matrix: new Float32Array([1, 0, 0, 0,
0, 1, 0, 0,
0, 0, 1, 0,
1, 2, 3, 1]) // the translation here is the camera's
// center of attention
},
{
type: "set_property",
path: "/Cameras/default/rotated/<object>",
property: "position",
value: [2, 0, 0] // the offset of the camera about its point of rotation
}
To move the camera itself to the point [1, 2, 3]
and lock its controls, we suggest setting the transform of /Cameras/default
to the exact camera pose and setting the position
property of /Cameras/default/rotated/<object>
to [0, 0, 0]
:
{
type: "set_transform",
path: "/Cameras/default",
matrix: new Float32Array([1, 0, 0, 0,
0, 1, 0, 0,
0, 0, 1, 0,
1, 2, 3, 1]) // the translation here is now the camera's
// exact pose
},
{
type: "set_property",
path: "/Cameras/default/rotated/<object>",
property: "position",
value: [0, 0, 0] // set to zero to lock the camera controls
}
The MeshCat javascript sources live in src/index.js
. We use webpack to bundle up MeshCat with its Three.js dependencies into a single javascript bundle. If you want to edit the MeshCat source, you'll need to regenerate that bundle. Fortunately, it's pretty easy:
If you have Docker + BuildKit installed, then you can run rebuild/rebuild.sh
to hermetically regenerate the dist
files. The very first time you run it, it
might take a little time to download all of the relevant files. Subsequent runs
should be much faster (~5-10 seconds).
This option offers the easiest way to rebuild, with the drawback that "watch" mode of webpack (to automatically rebuild every time you save) is unavailable.
To install Docker + BuildKit on Ubuntu, run:
sudo apt install docker.io docker-buildx
To run as non-root, you will probably also need to update the docker
group
permissions:
https://docs.docker.com/engine/install/linux-postinstall/
This option offers a convenient "watch" mode during an edit-test development cycle, but if you're not careful when installing the yarn+node+npm tools you might damage other software on your computer.
node
and npm
. Try running yarn -v
and npm -v
to make sure those programs are installed.yarn
project.json
file and install all the necessary javascript dependencies.npm run build
dist/main.min.js
. The build script will also watch for changes to the MeshCat source files and regenerate the bundle whenever those source files change.main.min.js
in your own application, or you can open up dist/index.html
in your browser.Note that due to caching, you may need to do a hard refresh (shift+F5 or ctrl+shift+R) in your browser to reload the updated javascript bundle.