Querying panoptic information for points

[leonard-git-acc]

Hello, I am currently trying to query panoptic data from the map for given points. To do this task, I am using a modified version the PlanningInterface::getDistance method:

...
      float sdf;
      voxblox::Interpolator<TsdfVoxel> interpolator(&(submap.getTsdfLayer()));
      if (interpolator.getDistance(position_S, &sdf, true)) {
        if (is_free_space) {
          current_distance[2] = std::min(current_distance[2], sdf);
          found_submap[2] = &submap;
          observed[2] = true;
        } else if (submap.isActive()) {
          // Active submaps reconstruct everything, take highest resolution
          // observation. Lower resolution observations are filtered before.
          current_distance[0] = sdf;
          current_resolution = submap.getConfig().voxel_size;
          found_submap[0] = &submap;
          observed[0] = true;
        } else {
          // Inactive submaps capture only their own geometry, return min.
          current_distance[1] = std::min(current_distance[1], sdf);
          found_submap[1] = &submap;
          observed[1] = true;
        }
      }
....

I am basically saving a reference to the submap, that the distance information was taken from.

// Aggregate result.
  for (size_t i = 0; i < 3; ++i) {
    if (observed[i]) {
      info.signed_distance = current_distance[i];
      info.class_label = found_submap[i]->getClassID();
      info.instance_label = found_submap[i]->getInstanceID();
      return true;
    }
  }
  return false;

In the end i try to get the class and instance id from the submap. Is this a good way to go about doing this or is there a better solution, to get panoptic data for a single point from the map?

Thank you for your help in advance.

[Schmluk]

Hi @LeonardRabes

Thanks for reaching out. Querying the panoptic information is in fact a bit complicated if you want to do it properly, mostly because the system can be configured quite flexibly.

The way the distance is queried is order by the highest reconstruction accuracy (highest resolution submap), so your implementation would give you the label of a high-resolution active map most likely.

On top of my head there are two main ways of doing this:

1. Conceptually, labels are only meaningful on surfaces. So you could return the label of the submap whose surface is closest (lowest sdf). For submaps with class layer make sure to check whether the queried voxel is considered to be belonging to the submap, otherwise you may get a wrong label (for each submap more geometry than strictly belongs to it is captured).
2. The theoretically most correct way of doing it is gathering all submaps for which the point is close to a surface (i.e. within the truncation band), getting their labels and class voxel belonging probabilities in a list, and then probabilistically fusing all these records.

I hope this helps! If you implement a nice solution for this please feel free to open a PR, I figure other people would also find this useful. I also added a bit more description and fixed a minor typo in #68, maybe that helps.

[leonard-git-acc]

Hi @Schmluk thank you for the input, your second suggestion is a lot better, than what I had until now. Even though I could only fuse the voxel belonging probabilities with the signed distances (in conjunction with truncation distance). So far I have been unable to access or initialize the score layer (I assume it contains the label probabilities) and I couldn't find it set up in any of the config files. Is there a way to use the score layer with multi-TSDF setup?

[Schmluk]

What do you mean by fusing the probabilities with the distances? The geometry and segmentation are independent. The segmentation information is stored in the class layer (not in the score layer, this was added to store arbitrary scores, e.g. uncertainty estimates). Depending on which segmentation/fusion method is used, the class voxels store only a single probability or a distribution. In either case, the probability of being part of the submap can always be queried using ClassVoxel.getBelongingProbability(). Hope this helps!

[leonard-git-acc]

I got a bit ahead of myself there, sorry about that. I tried to combine both approaches you named earlier. I calculated a relative distance from the truncation distance and the signed distance and multiplied it by the belonging probability of the voxel. I wanted to create a metric, that takes belonging and distance both into account:

// relative distance from trunc dist and signed dist
const float rel_dist = 1.0f - std::min(std::abs(signed_dist)/max_trunc_dist, 1.0f);
// calcualate metric for both distance and belonging
const float metric = belonging_prob * rel_dist;

Do you think this is a viable approach? Otherwise I will just use the belonging probability and search for the highest value at the queried point, but only if the highest value voxel determines itself as part of the submap.

I understood wrongly, that your second approach needs label probabilities and class voxel belonging probabilities and I thought the score layer contained these label probabilities. That doesn't seem to be the case for multi TSDF mapping.

[Schmluk]

Sounds very interesting! Generally both methods should work, with perfect segmentation I would expect the closeness and the belonging probability work well. Combining both might also work. It's a bit hard to predict how well each method would capture the semantics with noisy predictions, would be very interesting to study this experimentally! Let me know in case you come around to test this.

Regarding your second point, while currently no label probabilities are considered that is something that of course could be added to each submap, updated in the tracking stage (after associating detections to submaps), and included in the fusion process.

Hope this helps!

[Schmluk]

Closed due to inactivity.