[geometry] Mesh and Convex shape specification uses in-memory mesh

[SeanCurtis-TRI]

Problem

Currently, when specifying a shape for SceneGraph via Mesh or Convex, the mesh data is solely specified by a path to a resource on disk.

For applications such as cutting or doing on-line decomposition of an object or space, that generate meshes, it is unreasonable to demand that the mesh representation in memory be forced to go to disk in order to end up in SceneGraph. It would be better to go directly into SceneGraph.

Proposal

Modify Mesh and Convex such that they can take mesh data and take ownership of that directly.

(Edit: 10/15/21 We create a "universal" mesh type in Drake. It would carry all of the mesh information we typically care about: vertex positions, normals, texture coordinates, materials (including textures), etc. It would also support queries about the fundamental nature of the mesh: is it convex? Is it closed? Etc. Finally, we create instances of the universal mesh from the various forms of files we parse as well as allow programmatic instantiation.)

Challenges

1. We need to broadcast meshes to visualizers when there are no files on disk. Instead, we have to broadcast the mesh data directly.
   • (Edit: 10/15/21 This is not so problematic. It has been done to a limited degree in #15773 and #15772 and will soon be added to the C++MeshcatVisualizer. These only include vertices and normals, but texture coordinates could definitely be added. Textures would be the only unknown quantity.)
2. We need an in-memory representation that contains at least as much information as we'd get out of our currently supported file formats (e.g., obj). This includes vertex positions, normals, texture coordinates, groups, material information, etc.
3. Whatever the representation that is owned by Mesh and Convex, we'll need to translate from it to the various engines we currently support (RenderEngineVtk, RenderEngineGl, ProximityEngine -- aka FCL).

[jwnimmer-tri]

Here's a proposal for a quick win that I suspect would help a lot of people, be quick to implement, and step towards the goal.

The Convex currently takes a filename -- the named file's contents can be whatever your particular reifier knows how to deal with. Things like hydroelastic proximity just read the *.obj and ignore the material properties, don't support dae or vtp, etc.

Even if we create a first-class datatype that fully captures a mesh (as an array of vertices, normals, etc.) for interrogation directly in Drake C++, I think we still want to allow for the possibility for SceneGraph to just be a file router instead of a file parser. If I want to give some mesh data to my particular custom reifier, I don't need SceneGraph to parse it, I just need it to associate my data with the geometry id, and regurgitate it when asked.

Therefore, in the same way we currently can pass in a filename as the representation of a mesh shape, we could allow passing in file contents as the representation of a mesh shape. As a starting point, this would let current users ditch their tempfiles as part of the cycle. They are already turning their data into an obj string, but now we can eat that string directly.

There's still a good use case for having a first-class, in-memory model available to save the cost of parsing ascii numbers all over the place for code that wants to program against a custom Drake datatype, but ditching filenames for in-memory strings would still be (in my estimation) a huge win.

[SeanCurtis-TRI]

I like that.

In the same way that a known geometry can get a property wholly unknown to SceneGraph, but SceneGraph nevertheless promises to propagate the property and make it available, SceneGraph could take an otherwise unknown shape type: GenericShape. One could assign properties to it, afix it to a frame, etc. SceneGraph won't do anything with it (i.e., won't appear in Drake-defined renderers, proximity queries, etc.) except return it as a result if you ask about shapes in the world, or shapes affixed to a frame, or shapes with a particular role.

I would hypothesize that its data would be AbstractValue, such that geometry data of arbitrary shape could be jammed in: a string consisting of file contents, a fully developed struct defined by the user, etc. They simply need to be able to safely attempt to pull their data back out (e.g., AbstractValue::maybe_get_value()).

This unique type helps distinguish it from every other Shape type which is explicitly known by SceneGraph and have a consistent domain of support.

And, in the future, we could even provide some form of callback such that a user could provide a parsing function that turns their GenericShape into a Drake-known Shape so that it can fully participate in all of the SceneGraph controlled operations.

@jwnimmer-tri the big question is: what are the work flows you have in mind that this would facilitate? The one I have in mind is the ability to affix arbitrary visualization artifacts to SceneGraph frames and have a custom visualizer be responsible for drawing them. Is that yours? Do you have others?

[SeanCurtis-TRI]

@RussTedrake the above GenericShape may be a better solution for your Meshcat issues than MeshcatCone, MeshcatMesh. It would empower your visualizer to increase the range of geometry it supports independently of SceneGraph. Thoughts?

[RussTedrake]

@SeanCurtis-TRI I've already got MeshcatCone ready to PR, and I think having a first-class Shape type for Cone makes more sense. It is a perfectly reasonable primitive with a simple constructor taking (up to) 3 values.

I'm trying to think through exactly what the GenericShape workflow would look like. Presumably, in my ShapeReifier, I would have something like

void ImplementGeometry(const GenericShape& generic, void* data) {
  if (generic.value.type_info() == typeinfo<MeshcatCone>()) {
     const auto& cone =generic.value.maybe_get_value<MeshcatCone>();
     ...
     // proceed with what I currently do in ImplementGeometry(MeshcatCone..) 
  } elseif (.. MeshcatMesh...) {
    ...
  }
}

And I would have to wrap the logic to create the AbstractValue of the MeshcatCone.

I think at that point, I would just be better off supporting a separate entry point for SetObject(path, meshcat_cone, rgba) and forget about trying to jam myself into the Shape hierarchy.

[RussTedrake]

FTR -- I do think that @jwnimmer-tri 's suggestion of having the obj string in memory instead of the filename would work for me in the short term, whether you implement it as GenericShape or not. (They seem like different ideas)

[jwnimmer-tri]

I agree with Russ that my idea of "pass mesh file contents as data buffers, instead of as filename pointers" is distinct from Sean's idea of "abstract-valued geometry hub". They share the same kind of inspiration -- that SceneGraph should get out of the way (or at least be less prescriptive) when geometry producers just want to pass along geometry data to geometry consumers -- but in practice the two are quite different (and likely complimentary) approaches.

For the mesh data buffer in particular, I think that the filename-specific shapes that we have now (Mesh, Convex) probably need to grow a bit less prescriptive. Instead of thinking in filenames or data buffers, we should probably think in terms of URI:

• If the URI is file:// then we've subsumed what we have now.
• If the URI is data:// then we can have inline data.
• If the URI is package:// then we use the ROS package path to resolve the contents.
• If the URI is https:// then we can fetch resources from the network.

When parsing a SDFormat or URDF file, if it uses package:// for meshes (as will almost always be the case), ideally we should preserve that package URI for downstream consumers to act on, rather than immediately resolving it to an absolute path and throwing away the URI. However, for geometry consumers that actually want the absolute path or file contents, there should be an easy way to ask the Shape (or a Shape-adjacent class, e.g., QueryObject) to provide the filename or file contents, so that the kinds of calling code we have today don't become unnecessarily complication.

The other missing concept is mime type. If we are going to pass around data without an associated filename (+ extension), we should probably be able to use one of the well-known mime types to identify what kind of mesh format is in use.

It's possible that we want to have multiple URIs available for mesh data (with equivalent contents), so that the geometry consumer can choose whether they want to use the package URI, or file URI, or http URI.

I haven't tried to pencil out how all of those ideas turn into a C++ API for shapes that require supplemental files, but I think that any solution should be framed in terms of those ideas. Even if we want a land quick fix to get objs-as-string-data into SceneGraph, we should do it in a way that leaves room for the larger design space.

[SeanCurtis-TRI]

As I re-read this (much later), I took an obnoxious tangent.

The original topic (and what @jwnimmer-tri addressed) was simply: how do I instantiate a Mesh when the data is in memory and not on disk. The question of "parsing", "getting in the way", and simply passing geometry data from a producer to a separate consumer are red herrings. This issue is about creating a mesh in memory and having the end result be just the same as if it had been parsed from a file.

I went off on the tangent about SceneGraph's ability to agnostically transport data and turned that into GenericShape. I must've had the MeshcatCone on the mind when I wrote that, but it really has nothing to do with the original issue.

So, I'm bringing this back to the original purpose: go from memory to Mesh/Convex in SceneGraph.

To that end, it seems to me that if we did nothing but bring in the data:// URI prefix for the Mesh/Convex data, we've opened the door for the other URIs. In initial pass we could assume the mime type and, again, add that in the future to allow greater distinctions (that would go well with the increase of the mesh file formats that SceneGraph accepts).

The only downside there is the necessity to translate binary data to a string, and then parse it back. It won't be the most efficient way to do on-the-fly geometry changes.

[jwnimmer-tri]

(Aside: I've recently been reading that push for calling them URIs has apparently petered out; and most people think we should just say URL again now. I'll use that terminology in my reply here.)

I think at the same time we add in-memory unparsed data, we do need to add a mime type from the get-go. I don't think it will be that difficult, and I think it's important to steer away from "sniff the first 1kb and take a guess" heuristics that will inevitably occur without a mime type.

Note that our use of URLs in our docs/classes/functions as a tool to explain the concept does not necessarily imply that we need to base64-encode our unparsed meshes while they sit at rest in memory. Nothing is stopping us from storing the file contents directly, and providing a Mesh function that returns the unparsed byte string, or accepting an unparsed byte string (and mime type) on the way in. Only if we need to serialize the URL for out of band use would be need to explicitly encode it into the canonical URL format. The data::// URL is just modelling abstraction; the actual implementation can be as efficient (or inefficient) as we like.

[SeanCurtis-TRI]

Note: A first foray abusing the notions of URLs was done in #21581. That effort was dismissed in favor of an alternate approach.

Aside from the fact that it was very much abusing URL nomenclature, that approach really limited our mesh types; it precluded specifying a mesh that uses a menagerie of files. Instead, we're prototyping a version where a family of files can be provided. It needs to be tested against multiple uses in Drake to make sure access to the data is sufficiently convenient to not be a pediment to any non-TRI Shape consumers.