RenderRpc: Evaluate and select the open-source 3D scene description format

[zachfang]

Evaluate different open-source 3D scene formats and select the most promising one for the initial implementation. The decision can be revisited later on if needed.

[RussTedrake]

I don't know how close we can get with sdformat, but it seems to me that it would be a huge win if that could suffice.

[jwnimmer-tri]

It is an interesting idea, but @RussTedrake could you elaborate a bit more on why serializing a SceneGraph into SDFormat for rendering would be a win? We might be missing part of your thought process.

So far, I don't immediately see any win from the Rendering RPC point of view alone.

The only win I can see is that, as a side effect, we would have some library code that could save a SceneGraph into SDFormat in the middle of a simulation. For that, we'd presumably also want to incorporate the MultibodyPlant into the serialization routine, so that we preserve the kinematic tree? (By default, rendering doesn't care about the kinematics, only about the absolute geometry poses at some instance.) There would also be some small added complexity around also serializing the proximity or illustration geometry, instead of only the perception geometry.

One proposal floating during the kickoff meeting was to send the scene as gLTF. As I understood it, that had the benefit of being well-tuned to express the kinds of information that a render server would care about (lighting, PBR textures, etc.), as well as (usually) already having built-in parsing support within most existing renderers. Maybe the render server could load the environmental scene from gLTF, but the robots & manipulands could still come in via RPC as SDFormat.

[ToffeeAlbina-TRI]

Formats to evaluate: glTF USD SDFormat

Criteria:

• what's the feature set - what can be rendered, what is supported
• what is the library support for dealing with this - how much code do we have to write/maintain vs already exists
• is it self contained or not - might we need cross-process access to arbitrary filesystems
• ... TBD

[ToffeeAlbina-TRI]

TRI to create a scene in drake that exercises what we already can do in vtk in-memory - what we can currently do in drake

AND create a glTF file of the homecart scene which shows the end goal that we want

this will force TRI to come up with the enumeration of the minimum features we want to cover

[ToffeeAlbina-TRI]

Fundamental api calls are intended to do:

• add geometries
• update poses in the scene
• image of current state of affairs

[sherm1]

Now that MuJoCo will be open source, would it make sense to consider its MJCF scene description language also?

[SeanCurtis-TRI]

My general suspicion is that formats like SDF, URDF, and MJCF are impoverished when it comes to data necessary for rendering.

[zachfang]

TRI to provide two working examples as the baseline and ultimate evaluation criteria.

• [ ] A glTF file contains all (or most) of the rendering features TRI would like to express ultimately for a rendering scene.
  • PBR texture
  • Lighting configuration (point light, spotlight, area light*, global illumination)
  • Transparent, translucent objects
• [ ] A Drake RenderEngineVtk example(s) serves as the baseline about what we currently can do.
  • Different shape geometries
  • With and without texture
  • Properly map a given texture
  • Logic to load a texture (from OBJECT_NAME.png rather than OBJECT_NAME.mtl)
  • Camera configuration (extrinsic, intrinsic)
  • Moving scene rendering
  • Clipping range for both color, depth, and label images
  • Depth range in a depth image
  • Exercise five different types of labels (empty, don't care, don't render, unspecified, user defined) in a label image
  • (optional) Layered transparency

[ToffeeAlbina-TRI]

Toffee verifying we can shared blender file with Kitware

[zachfang]

We have spent some time evaluating different options, and glTF is the most promising scene format for our needs. Thus, our initial prototype is designed to use glTF to send scene descriptions between a client and a server.