models: Should find automated workflow of importing articulated robots (URDF / SDFormat) from other ecosystems

EricCousineau-TRI commented 3 years ago

This is a narrowed version of #14436; creating per VC this Thursday (2021-05-06) w/ @azeey and others in SDFormat project.

I've tried doing some really simple conversions here: https://github.com/EricCousineau-TRI/repro/tree/ea2d7987c/drake_stuff/mbp_robot_arm_joint_limit_stuff https://nbviewer.jupyter.org/github/EricCousineau-TRI/repro/blob/ea2d7987c/drake_stuff/mbp_robot_arm_joint_limit_stuff/joint_limits.ipynb

Initially, it weren't too bad, until DAE-to-OBJ conversion failed or had rough edges, at least with the basic steps that I tried with pyassimp: https://github.com/assimp/assimp/issues/3703 https://github.com/assimp/assimp/issues/3705

Acute goals:

Document a working and automated solution to consume things like URDFs from ROS/ROS2 ecosystem, and pull in robot arms, quadrotors, etc. like Ignition Fuel.

Non-goals:

Integration Ignition Fuel fetching into Drake itself. (That should be a separately-scoped issue.)

\cc @sammy-tri - having encountered similar issues \cc @SeanCurtis-TRI - first step is to find min-effort solution; maybe a follow-up would be to see what is necessary to resolve #14436 / #2941 \cc @IanTheEngineer - regarding ROS ecosystem

jwnimmer-tri commented 3 years ago

... solution to consume ...

I'm unclear on the victory condition here -- what does "consume" or "pull in" mean? Is it as simple as "Parse into MultibodyPlant and SceneGraph without throwing any exceptions, and then display it in drake-visualizer"?

EricCousineau-TRI commented 3 years ago

Sorry - yup, that it parses, loads, and is correct. In the above workflow, it wasn't correct b/c converted meshes lost an internal transform.

marcoag commented 3 years ago

I faced and verified these bugs on the bionic and the pip versions of pyassimp:

https://github.com/assimp/assimp/issues/3704 https://github.com/assimp/assimp/issues/3705

These issues do not seem to happen when using the c++ version of the library or a later version of the python wrapper. Also, I faced the issue where collada files get rotated by 90 degrees when imported:

https://github.com/assimp/assimp/issues/849

I managed to work around this by setting the root node matrix to the identity matrix. Using both, the c++ and updated pyassimp versions I manged to generate the obj files that seem to get us in the right direction:

I guess there's two ways to go about this for now:

We use a C++ binary to convert the files
We move to focal or newer for a new system version of the python wrapper
We use the latest version from source

On a side note, the sanity check that the code was doing had a typo. After this correction it seems to fail due to changes (i.e. number of meshes) happening during conversion, so maybe we should think on a rewrite of this check?

EricCousineau-TRI commented 3 years ago

[...] that seem to get us in the right direction:

Just to check, is this the expected rendering of the given configuration? (zero configuration or otherwise?)

We move to focal or newer for a new system version of the python wrapper

This would be my ideal!

marcoag commented 3 years ago

[...] that seem to get us in the right direction:

Just to check, is this the expected rendering of the given configuration? (zero configuration or otherwise?)

That seems to be the case:

We move to focal or newer for a new system version of the python wrapper

This would be my ideal!

Roger that!

marcoag commented 3 years ago

The branch focal of: https://github.com/marcoag/repro is now working on ubuntu focal, you only need to do:

$ ./setup.sh
$ source venv/bin/activate
$ jupyter-notebook

And you can execute the cells in the notebook to get the model rendered as per the picture above.

@EricCousineau-TRI let me know if you want me to PR somewhere and any next steps for this.

EricCousineau-TRI commented 2 years ago

Thanks! I cherry-picked https://github.com/marcoag/repro/commit/3a86ebcf1437b2f283df8a702729ff06e9ffd974, but have not taken in Focal changes just yet.

EricCousineau-TRI commented 2 years ago

For next steps, can you try loading these articulated models from Ignition Fuel? (the process of downloading them isn't as important to me as just trying to import the assets into Drake)

marcoag commented 2 years ago

Current release of assimp fails to handle empty XML tags. Even though this was fixed already here: https://github.com/assimp/assimp/pull/2727, the latest release is older than the fix and it is not included. Some of the collada files from fuel have some empty information tags at the top, i.e.:

<COLLADA xmlns="http://www.collada.org/2005/11/COLLADASchema" version="1.4.1">
  <asset>
    <contributor>
      <author/>
      <authoring_tool>FBX COLLADA exporter</authoring_tool>
      <comments/>
    </contributor>
    <created>2021-03-22T21:23:15Z</created>
    <keywords/>
    <modified>2021-03-22T21:23:15Z</modified>
    <revision/>
    <subject/>
    <title/>
    <unit meter="0.010000" name="centimeter"/>
    <up_axis>Z_UP</up_axis>
  </asset>
  ...

They mainly seem to be there due to unfilled extra information that is not very relevant to the functionality of the models. For now, I added a workaround that will clean them prior to importing them with pyassimp. Just wanted to document this here in case anyone knows of cases where empty tags might be useful and need to stay.

EricCousineau-TRI commented 2 years ago

Per VC, I'm good for us using the workaround to adjust for empty XML tags, assuming it's scripted, just with a TODO citing the PR you mentioned, e.g. TODO: Remove this once all consumers have assimp release incorporating assimp#2727 available on their distribution.

marcoag commented 2 years ago

I managed to make all the models work in the code here. The current workflow is as follows:

The user downloads a model from fuel.
Provides the directory and model file name to the script. 2.1 If the model file is an SDF file, workarounds to make them work are applied (see below for the latest additions on this). 2.2 Otherwise the previous workflow of downloading the universal robot is followed.
The model can be used (I tested it by loading on the jupyter-notebook).

A few notes on the changes that I added (only for sdf model files):

MPL right arm contains a comment before the xml prolog. This seems to produce an error in parsers and seems to be just wrong. I reported it and it will be changed. However as a precaution and since this does not affect the model I am currently removing the xml prolog before parsing.
Some models from fuel contain materials that point to specific Gazebo scripts and fail to be parsed in drake. As a workaround I remove all references to a Gazebo script that I find in the sdf file.
Some models use package based uri and do not contain a package.xml. In order to allow drake to find them I create a package.xml with the model name before parsing so it finds the meshes.

A few open issues:

I am not too sure on how to handle the 2.2 part of the workflow, I guess we should consider urdf files there (anything else?). The current code is specific to the universal robot repo but I don't know what other files we expect to encounter in case we want to generalize this code.
~~Currently the sanity check after meshes conversion is commented as it doesn't work. I'm in the process of figuring out why and/or working out something else. Any suggestions are welcome :)~~
I haven't tested the models further than just loading them on the jupyter-notebook in the repo and checking they look the same as when loaded in gazebo. Any further tests we could perform? Any automated ones?

marcoag commented 2 years ago

Sanity check is fixed now. I will continue working on further checks probably using Intersection over Union in combination with another tool...

marcoag commented 2 years ago

It seems like models are currently loaded with a single diffuse color and textures are ignored. I noticed some models come with jpg files that would be loaded just by converting them to png as long as the name is the same as the mesh file.

From https://github.com/RobotLocomotion/drake/issues/11949 it seems that there's not much more support than the png loading on the drake side... should we consider textures as part of the work of this issue or shall we leave it for future work and maybe wait until there's more work done on the drake side?

EricCousineau-TRI commented 2 years ago

Yes, texture loading should ideally be covered as part of this.

For now, we should include image conversion (case-insensitive, *.jpg, *.jpeg, perhaps *.tiff?) to *.png. For the future, we should consider natively supporting other formats. @SeanCurtis-TRI Is this worth noting in an issue?

EricCousineau-TRI commented 2 years ago

Per f2f:

Should enable viewing + comparing collisions
Should check num_actuated_dofs vs. num_positions

I'll provide some additional example URDFs.

SeanCurtis-TRI commented 2 years ago

@EricCousineau-TRI

Is this worth noting in an issue?

I think it's worth noting in this issue. When @jwnimmer-tri asked what the vectory conditions were, you responded agreeing with what you said but added the word "correct".

From reading this issue (particularly the main description), I have no idea at all what constitutes correct. I suspect that if the source files have diffuse textures, then correctness includes that the result of our parsing operation should include a model visualized with the same texture. How that is achieved depends on what the source files are and what the pipeline is. (The simplest is to a conversion to an (obj, mtl) pair. That should parse in drake_visualizer with texture.)

For now, we should include image conversion...

To what end? If it's strictly for drake_visualizer, I suspect that's not necessary. If it's for the perception role as well, it might be necessary. Again, without clear understanding of the victory conditions, it's difficult for me to assess the statement.

jwnimmer-tri commented 1 year ago

Note that as of #9498 which just landed in the past two weeks, we now have PackageMap.AddRemote which can download remote archives on demand.

What seems to remain here is helping with incompatibilities between the existing data that users can fetch, and the kinds of data that Drake can load. Also writing a tutorial.

jwnimmer-tri commented 1 week ago

This ticket was basically a project planning ticket for OSRC, and that contract has now expired.

For incorporating DAE-based assets into a Drake simulation, we have https://github.com/RobotLocomotion/drake/issues/19109 as a nice tracking issue.

For incorporating STL-based assets into a Drake simulation, we have https://github.com/RobotLocomotion/drake/issues/19408 as a nice tracking issue.

There is also https://github.com/RobotLocomotion/drake/issues/14436 which is related in terms of file format support.

There is also https://github.com/RobotLocomotion/drake/issues/20444 for importing MuJoCo assets easily, which as of today is a more important hotness than legacy SDFormat and legacy-squared URDF.

In any case, this ticket is not particularly actionable since there is no longer any contract to support it, so closing as "not planned".

RobotLocomotion / drake

models: Should find automated workflow of importing articulated robots (URDF / SDFormat) from other ecosystems #15024