Tracking issue: SDF Support in rmf_site

[luca-della-vedova]

This issue tracks what's been done and what's currently missing for SDF support in rmf_site_editor, to make it a full replacement for the legacy rmf_traffic_editor workflow.

Why

By adding a capability in site editor to export SDF models and worlds, we can integrate it with our current pipeline and start to get some real usage and feedback from it. Users can use rmf_site to have an accurate 3D preview of what their world would be like then export it to Gazebo.

What

The plan is to have a minimal implementation of the features offered by rmf_traffic_editor to build functional sites, as well as what is offered by rmf_building_map_tools to export them to Gazebo worlds.

How

There are three large parts to this effort, ordered by increasing controversiality. 2) and 3) are split in two different branches if we decide to not go for 3).

1) Feature parity with traffic editor. 2) Export SDF functionality, branch luca/export_sdf 3) Headless mode and world export, branch luca/headless_sdf_export_poc

Feature parity with traffic editor

This is the least controversial and area that we should be working on regardless. We need to make sure that all the features offered by traffic editor are offered by site editor, and the ones that are not offer a good migration path. On the top of my head, areas to check:

• [ ] Navgraph export: As of #201 most features are there but at the very least mutex is still pending. As mentioned in the PR some features might need validation as well.
• [ ] Laserscan alignment: Traffic editor offered a nice way to align laser scans to base drawings and show the relative transform between robot coordinates and RMF coordinates. It's not super critical but a nice to have.
• [ ] Support for WGS84 maps: We have an example in global coordinates in rmf_demos, the campus, that currently fails to import.

Export SDF

This is where the bulk of the work lies and where I took some decisions on how to approach it. A large part of the problem is how do we export the meshes for what we see in the world?

In the legacy workflow, we used to add include tags for each model in the world and manually generate obj meshes for items such as floors and walls by doing the triangulation, calculating the texture coordinates etc. This works but in my opinion is not too optimal for a variety of reasons:

• When we start to have many static models, startup can take a long time (in my experiments it would reach 40-50 seconds).
• We need to maintain triangulation in two parts, when we generate the meshes in site editor and when we export them, and it might be easy to make mistakes that introduce a discrepancy between the two versions.
• Something similar applies to models, if for example we import an SDF but some features are not supported by site editor, when exporting it it might be different from what is shown (i.e. SDF with material properties will be rendered differently in site editor and Gazebo).
• In site editor we also allow a variety of assets. Not necessarily only SDFs but also local meshes. This would require a different strategy compared to <include> tags.

For these reasons I took the executive decision to rethink how we export meshes. I created a crate (warning it's very experimental!) to export arbitrary bevy meshes to GLB files. The idea is that we can take anything that is rendered and throw it in a GLB exactly as it is, so we are guaranteed to have parity between what we see in the site editor and what is exported. The site editor then collates all static meshes in each level into a single large file, and all non static meshes into separate files. This has a few benefits:

• Only one mesh generation path to maintain, we don't need to worry about (for example) generating wall and floor meshes from wall specifications the way we used to and whenever we introduce any change to how they are rendered this will be automatically reflected in the exported files. This already proved beneficial since for example we introduced automatic hole generation in #196. This was done as a manual step in traffic editor but now comes automatically with this pipeline.
• We can combine static meshes into a single large glb file and do some optimization there (that the crate is doing). For example, reuse vertex buffers / materials for similar meshes. This makes world models "fairly" small. Furthermore, since the static mesh is now a single object, simulation startup times are a lot better.
• We are guaranteed that what we see in the site editor is what we will see in Gazebo, so for example if we import a model with an SDF tag that is not supported in site editor, it will not be exported to Gazebo.

There are however some drawbacks:

• Materials are exported exactly the way they are rendered but in some cases this might not be desirable. For example, floors are set to semi-transparent by default when a drawing is added to make sure the users can see both. For this reason exported floors will be semi transparent which might not be what the user expects. I solved this by despawning drawings in headless SDF export mode. It's not ideal but worlds are not cloneable so the alternatives were cloning the materials, ovewriting the transparency or skipping the transparency updating system in headless mode.
• Same for meshes, I still have to test to be sure but I suspect the outlines might be exported if it was to be rendered while exporting. I checked and it seems outlines are not part of the mesh.
• Exporting the meshes is somewhat computationally expensive. This is mainly due to the deduplication done in the bevy_gltf_export library which uses a "crude" HashMap<Vec<u8>, Index> to reuse binary buffers between differen meshes without which the static meshes would be very large.
• SDF export and mesh export cannot be done in the same place. Specifically, rmf_site_format takes care of converting a site to an SDF world and this can be done regardless of the site editor. However, loading assets and spawning meshes requires a full site editor running, this part of the export is done in the site editor itself. This is especially tricky for meshes since rmf_site_format needs to export a world that refers to a specific mesh with a certain name, and the site editor needs to make sure it follows the same naming convention. This works for now but is a bit brittle.

In general I'm quite convinced that this is a good way forward but there are still rough edges. As I will discuss later.

Headless mode

The (current) idea is to make rmf_site_editor a drop in replacement for rmf_building_map_tools. If we analyze the map building pipeline we can see that map building is a several step process:

• Map building itself, including creating the SDF world and the meshes for floors and walls.
• Model downloading with pit_crew.
• Navgraph generation.
• Crowdsim update.

All of this is done at build time of the map package. The idea is that to replace this workflow we would need the site editor to also have the capability of being run in headless mode and export the world as an SDF.

This is the part that is most hacked together right now. In my POC implementation I added a (non wasm) command line argument --headless-export [file_name]. When the site editor is ran with it, it will not initialize a window and run until all the models either have spawned or failed spawning, finally it will send a save event and save the whole package including world meshes and navgraphs to the requested path. This is a bit hacky however, and not really CPU efficient since I believe most of the time the application is I/O bound and doesn't need to actually do much since it's fetching models. It currently seem to take a quite significant amount of time but I'm not exactly sure why. I suspect it's because for each model it needs to try to fetch them from fuel in a few different formats and that can take some time.

Current caveats and rough edges.

• [ ] The biggest one, collada meshes are not supported. This means that some worlds that use them will fail to load and export them (the airport world is one of them). In my implementation I made this a silent failure so the world will just be exported without the mesh but it can be upgraded to be an error instead. I personally don't believe it is worth it to support collada since it's an abandoned format and Gazebo supports gltf since Garden and there will be Harmonic (LTS) out soon with gltf support as well. I would actually advocate for slowly transitioning fuel models to gltf but that is just my opinion.
• [x] The second biggest one, we skipped Gazebo Garden in Open-RMF and this features targets Harmonic straightaway. However, as of the time of writing, Harmonic binaries are still not available so it needs to be built from source for testing. This will fix itself with time.
• [x] This PR also uses the ECS based refactor on rmf_simulation, mostly to avoid having to write it for an outdated stack.
• [ ] The airport terminal world runs at a terrible real time factor which doesn't seem to happen with other worlds. I saw an error from the physics engine Trimesh-trimesh contach hash table bucket overflow and I believe the static collision mesh is too big. But anyway airport terminal shouldn't be our main demo since it also includes a lot of collada assets.
• [ ] Since dispensers and ingestors are just models in the legacy workflow they wouldn't quite work with this new one (we drop plugin tags). I worked around it for now by manually adding an include tag whenever the model name is either TeleportIngestor or TeleportDispenser but this is obviously not ideal and we should get rid of it. We should have proper support for these anyway at some point so we can revisit once there is a concept or an ingestor, dispenser, or in general workcell in the site editor.
• [ ] Similarly for robots. For now I just use an include tag but ideally at some point we would have concept of a robot in the site editor, at which point we can use those properties to generate a proper tag.
• [ ] The reliance on hard coded names between rmf_site_format and rmf_site_editor is not ideal, maybe it could be fixed if we had an API that returns the "sdf" names for some elements such as doors and lifts.
• [ ] I had a lot of trouble with the default camera pose, it seems I had to flip the signs of bevy quaternions when bringing them into gazebo, in the end I just hardcoded the rotation the same way it's done in the legacy pipeline.
• [ ] The code itself is a bit messy since world files are complex beasts. It is especially tricky to work with <plugin> tags since their serialization / deserialization must allow for arbitrary trees so we need to bypass / lose a lot of the benefits of serde. At the end of the day the amount of code added is still less than the size of rmf_building_map_tools with almost the same functionality, but still.
• [x] Finally, since the data format in rmf_site is different from the legacy .building.yaml and it uses ron, it was necessary to create a new node for serving the building map. This node uses rclrs so it also needs rclrs to be compiled and sourced. Solved by switching to json.

Test it!

As mentioned in the caveats testing is a bit involved right now. Specifically you will need to:

• Build and source Gazebo Harmonic.
• Add rmf_site to your workspace, branch luca/headless_sdf_export_poc.
• Checkout the following branches:
  • rmf_demos: luca/site_editor_poc
  • rmf_simulation: luca/ecs_plugin_sandbox
  • rmf_traffic_editor: luca/site_editor_support
  • rmf_ros2: luca/event_based_updates
  • rmf_visualization: luca/event_based_states
• Add ros_gz branch iron to your workspace.
• Make sure to run export GZ_VERSION=harmonic in your terminal to make the ros_gz_bridge compile against harmonic
• Do a clean build.
• Run the demos as usual.

It should work! I tested the following:

• Hotel world with the README tasks (one robot doing patrol between two floors and one robot cleaning).
• Office demo with delivery and patrol.
• rmf-web visualization and control of doors and lifts (latter pending https://github.com/open-rmf/rmf-web/pull/930) as well as trajectory visualization. Only missing part here is the visualization of named waypoints.

Note that the linked rmf_demos branch switches the whole pipeline (world building, as well as building_map_server) to use .site.json files. In the branch itself I converted and committed office.site.json and hotel.site.json so these two demos (and only these two demos) should work. Other worlds can be trivially converted but I avoided it in the interest of not committing too many files.

[mxgrey]

floors are set to semi-transparent by default when a drawing is added to make sure the users can see both

I wonder if it would make sense to extract a clone of the world and then directly override the components for these transparency settings, and then export that replicated world instead of the original? I think it's very unlikely that the user ever intends to have a glass floor.

[mxgrey]

I suspect the outlines might be exported if it was to be rendered while exporting

Same for this, maybe after we duplicate the world, we can delete all outline meshes from that world before exporting?

[arjo129]

The biggest one, collada meshes are not supported. This means that some worlds that use them will fail to load and export them (the airport world is one of them). In my implementation I made this a silent failure so the world will just be exported without the mesh but it can be upgraded to be an error instead. I personally don't believe it is worth it to support collada since it's an abandoned format and Gazebo supports gltf since Garden and there will be Harmonic (LTS) out soon with gltf support as well. I would actually advocate for slowly transitioning fuel models to gltf but that is just my opinion.

A third option would be to add a warning. This is what I've seen most software do for scenarios like this.

[luca-della-vedova]

floors are set to semi-transparent by default when a drawing is added to make sure the users can see both

I wonder if it would make sense to extract a clone of the world and then directly override the components for these transparency settings, and then export that replicated world instead of the original? I think it's very unlikely that the user ever intends to have a glass floor.

It's very simple to remove transparency from a floor then export it. I didn't do it because we do allow the possibility of setting transparent walls to support glass walls and I thought we might want ot keep the option open for transparent floors as well but I'll do that.

WRT the outlines, The bevy_mod_outline plugin uses a marker component (actually bundle) and adds a custom render pass that adds outlines to the mesh. Luckily it seems the render pass only changes the way the mesh is rendered and not its actual properties so it does not affect the output of the export. I'll update the first message to reflect that.

[mxgrey]

A third option would be to add a warning.

This seems like the perfect candidate for something that should go into the in-app log. Also the kind of thing that belongs in the diagnostic tool.

[luca-della-vedova]

The biggest one, collada meshes are not supported. This means that some worlds that use them will fail to load and export them (the airport world is one of them). In my implementation I made this a silent failure so the world will just be exported without the mesh but it can be upgraded to be an error instead. I personally don't believe it is worth it to support collada since it's an abandoned format and Gazebo supports gltf since Garden and there will be Harmonic (LTS) out soon with gltf support as well. I would actually advocate for slowly transitioning fuel models to gltf but that is just my opinion.

A third option would be to add a warning. This is what I've seen most software do for scenarios like this.

Yap we have warnings in place for this case! However, I believe warn! messages don't get propagated to stderr so are hidden by default when doing colcon build so they are not too helpful. eprintln errors do but we don't use those in rmf_site_editor so we can do better log filtering. I'll poke a bit and see if it is configurable.

[luca-della-vedova]

Test it!

As mentioned in the caveats testing is a bit involved right now. Specifically you will need to:

• Build and source Gazebo Harmonic.
• Build and source ros2-rust.
• Add rmf_building_map_server into your workspace.
• Checkout all the branches for the ECS based refactor and Harmonic migration in rmf_visualization, rmf_simulation, and rmf_ros2. Documented in Jazzy Jalisco checklist rmf#416
• Checkout the SDF POC branch in rmf_demos.
• Do a clean build.
• Run the demos as usual.

I managed to make this a bit less painful and removed the dependency on an rclrs node by doing the following:

• Adding support for saving site files as .site.json here.
• Adding a bare minimum json parser and publisher to rmf_building_map_server.

I am quite unhappy with the amount of duplicated code that this resulted in, but hopefully on the rmf_site side we should be able to cleanup a fair bit once we split the workcell editor mode out, and in the Python side we can either use rclrs when it's ready or slowly phase out the building yaml logic.

With these changes we can get rid of some of the steps and now the testing procedure becomes:

• Build and source Gazebo Harmonic.
• Checkout all the branches for the ECS based refactor and Harmonic migration in rmf_visualization, rmf_simulation, and rmf_ros2. Documented in Jazzy Jalisco checklist rmf#416
• Checkout the SDF POC branch in rmf_demos.
• Checkout the pointer rmf_traffic_editor branch.
• Add ros_gz and set the environment variable as pointed in the previous instruction.
• Do a clean build.
• Run the demos as usual.

This is for testing legacy projects. For new projects with the site editor, those should be saved as .site.json to make them compatible with the rest of the pipeline, not .site.ron. I'm open to changing the default but I didn't do it here since it would be a more controversial change.

And once Harmonic is released as binaries we won't need any special steps. If this looks reasonable I can start to open a PR and cleanup the code. Note that we should probably get around to doing more testing and reviews on the various branches of the ECS refactor for Harmonic / Jazzy.