Extract terrain heightmap

[diegoferigo]

I started considering the development of a plugin that extracts from the simulation the heightmap around a model over a equally spaced grid. A picture is worth a thousand words, from [1]:

Flat terrain


It could allow the integration of terrain-aware planning techniques without the burden of performing real perception, simplifying the prototyping process. Here below another image in a non-flat setting:

Stepping stones terrain


I'm a bit lost from where I can start, I'm not very familiar with the rendering part of the simulator. I guess that the visual elements of the world play a role. Can anyone with a greater experience on the rendering architecture please provide any hints about how the development of this plugin could be approached?

[1] https://arxiv.org/abs/1909.08399

---

I'm not sure how much this issue is related to #235 and #237, if it's a duplicate feel free to close it. It seems to me that this issue is about the inverse of #237.

[iche033]

Do you plan to use DEM or grayscale image to create the heightmap? If so, we need to support that first. Otherwise, if it's just regular 3D mesh, e.g. dae or obj, you should be able to get mesh vertices data directly from common::Mesh object once that's loaded into common::MeshManager

[diegoferigo]

In my naivety on the topic I thought it was possible to get this heightmap from any world location populated by any model.

If this is possible, the result I would be interested is a plugin that can be either a world or a model plugin. In the case of a world plugin, it would provide the map (x, y) -> z. If, instead, if inserted as a model plugin, it would provide the same data except that the model itself is ignored. A possible approach for improving the usability could be associating the map to a frame (e.g. a model link or any other coming from the new frame semantics) that can move in space, so that in both cases, world and model, the implementation would not differ.

I think this is the most generic heightmap that can be implemented, and it does not require any preparation of DEMs or grayscale images. I consider it as a local mapping feature that usually comes from the fusion of data from a complex perception pipeline (lidars, cameras, point clouds, and a lot of processing). I'm not sure whether this is doable and how much effort is required for such result. Again, naively I though that it could be done with an approach similar to how the lidar samples the environment, however I never used a lidar and never had a look to its implementation.

[iche033]

In the case of a world plugin, it would provide the map (x, y) -> z

I think this means you have some sort of heightmap in the environment already, and you need to be able to get the heights at given x, y pos? Some ideas: ign-rendering's RayQuery may be able to help with this. You can create a downward looking ray at (x, y) and compute the closest intersection point to get you z. Alternatively, you can place a downward looking depth camera to get the z values for a region.

[diegoferigo]

Alternatively, you can place a downward looking depth camera to get the z values for a region.

Thanks @iche033 for the hints! In the past weeks I started familiarizing with the rendering stack and now I think I have a better overview.

I explored ignition::gazebo::systems::Sensors and I understood that all the rendering sensors (camera, lidars, etc) are managed by this system. I successfully extracted data from a downward looking depth camera and what it provides is quite similar to what I originally had in mind.

The missing piece is now getting depth data from a downward looking depth camera excluding selected entities. If I want to get the terrain around a model, with this setup I have to exclude the model itself to be detected. Do you have any suggestions about how to do it?

I noticed that physics and rendering entities are constantly kept in sync, and the resulting Scene is then used by all sensors. A simple solution could maybe be creating a standalone System with an independent Scene just for this heightmap sensor, taking care of deleting every time given entities before updating the scene. I fear, though, that this approach would provide terrible performance since it would require a double rendering for just a single sensor. It would be nice having a brand new sensor that transparently provides this data, that could potentially be added to the existing Sensors system.

From your experience, do you see any workaround?