How to retrieve object name from the instance_id inside task_observations['gt_instance_ids']?

[antoniodecinque99]

I am trying to capture some images from the robot exploration in the scene. I want to create bounding boxes and object categories for each element seen in the ground truth instances. By using observations.task_observations["gt_instance_ids"] I correctly get the bounding boxes. However, such ids do not carry a mapping towards the object/receptacle class.

I basically have the same issue which was highlighted here: https://github.com/facebookresearch/home-robot/issues/431 I followed the solution, but I am not getting the correct classes. Any help would be appreciated, thanks!

[yvsriram]

Yes, it is possible that some objects do not have a mapping. Or are you finding that objects have incorrect mapping?

[antoniodecinque99]

I am finding objects which do not have a correct mapping to the task classes. Here's an example

In the ground truth I have objects with classes such as "magazine" or "potted_plant". I am doing evaluation of Detic and Grounded SAM to understand how to make such models better in the overall agent. How should I evaluate, for example, "potted_plant" if such class is not present in the complete vocabulary of task classes? Detic for example predicts "plant_container", which could be considered correct. How do I map such ground truths if they don't have an exact match in the vocabulary used for the task?

[yvsriram]

You need to map the object handles you get from the simulator to their categories. You can find the handle to category mapping here and here