RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.24k stars 1.25k forks source link

Support HDR environment maps in Meshcat blocked by `Meshcat` server #20366

Open SeanCurtis-TRI opened 11 months ago

SeanCurtis-TRI commented 11 months ago

The problem

As of #19837, Meshcat can specify an environment map. However, the images that are supported (png, jpg) can only support low dynamic range images. This has an appreciable impact on the illumination of the scene and underutilizes the power of environment maps in PBR.

Ideally, we should also be supporting typical HDR image formats (rgbe, hdr). These are images that encode not a normalized color space value at each pixel but a radiance value (an amount of light) at each pixel allowing for highly dynamic lighting effects.

meshcat uses three.js as the underlying graphics engine. three.js has an RGBELoader. However, it has been architected differently from three.js's TextureLoader which we use to load png/jpg images. The RGBELoader (via the DataTextureLoader) does not appear to accept urls with the data:/ prefix.

This means we can't simply marshall the base64-encoded bits of an image and stream them over to meshcat. We need to provide a server that will actually host the image.

I've submitted a question on the three.js form to confirm.

The solution

Drake's Meshcat class needs to serve as a host more generally for textures, meshes, etc. In addition to unblocking HDR environment maps, this will also allow browsers to enable their caches more intelligently.

jwnimmer-tri commented 1 month ago

For reference, our meshcat server can serve files now just fine, and indeed already serves the environment map over a http:// URL.

Maybe this is actually finished, except for a few API docs that mention HDR?