get_pb_ids_for_instance_ids returns wrong pybullet ids on macOS

[ChengshuLi]

@mjlbach could you provide your minimal example? @fxia22 would you mind taking a look?

[roberto-martinmartin]

I tested this. it was working some days ago. examples/observations/generate_additional_visual_channels.py should test and visualize it What is the issue? I see this:

[ChengshuLi]

I think the symptom is that the rendered ins_seg value doesn't match the pybullet body id, which should be the case by design.

[cgokmen]

Ins seg doesn't match body ID, it matches instance ID. Those are not necessarily equal. See InFOVOfRobot for how this is handled

[fxia22]

@ChengshuLi can you check if this solves your issue and close this issue if needed?

[mjlbach]

No, it's still not correct.

[fxia22]

@mjlbach what is the symptom of this error? Is this because mac uses non-optimized renderer?

[mjlbach]

The main symptom is that the values returned via body ids is wrong, so you cannot index into body_ids via the instance segmentation keys. For example, body_id 112 is missing:

Body ids via ins_seg: [ -1   0   8  10  68  78 112]
Body ids via scene: dict_keys([2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 54, 55, 56, 57, 58, 60, 61, 74, 78, 79, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95])

I am not using the instance id directly, but using the utility function to convert to pybullet ids:

body_ids = np.unique(env.simulator.renderer.get_pb_ids_for_instance_ids(
  state["ins_seg"]
))

Here's the repro, hit v to get the printout:

import os

import yaml
import numpy as np

import cv2
import igibson
from igibson.envs.igibson_env import iGibsonEnv
import matplotlib.pyplot as plt

def main():

    config_filename = os.path.join(igibson.example_config_path, "turtlebot_nav.yaml")
    cfg = yaml.load(open(config_filename, "r"), Loader=yaml.FullLoader)
    cfg['robot']['action_type'] = 'discrete'
    cfg['output'].append('ins_seg')
    cfg['output'].append('seg')

    env = iGibsonEnv(
        config_file=cfg,
        mode="gui_interactive",
        action_timestep=1 / 10.0,
        physics_timestep=1 / 40.0,
    )

    for _ in range(10):
        env.reset()
        for _ in range(500):  # 10 seconds
            # 15 seconds, StanfordVL/iGibson-dev#15 seconds split
            key = cv2.waitKey()
            if key == 119:
                action = 0
            elif key == 115:
                action = 1
            elif key == 97:
                action = 3
            elif key == 100:
                action = 2
            elif key == 113:
                env.close()
                quit()
            elif key == 114:
                break
            elif key == 120:
                action = 4
            else:
                action = 4
            state, _, _, done = env.step(action)

            if key == 118:
                plt.ion()

                _, axs = plt.subplots(2, 2, tight_layout=True, dpi=170)

                if "rgb" in state:
                    axs[0, 0].imshow(state["rgb"])
                    axs[0, 0].set_title("rgb")
                    axs[0, 0].set_axis_off()

                if "seg" in state:
                    axs[1, 1].imshow(state["seg"])
                    axs[1, 1].set_title("segmentation")
                    axs[1, 1].set_axis_off()

                if "ins_seg" in state:
                    axs[1, 0].imshow(state["ins_seg"])
                    axs[1, 0].set_title("Instance segmentation")
                    axs[1, 0].set_axis_off()

                body_ids = np.unique(env.simulator.renderer.get_pb_ids_for_instance_ids(
                   state["ins_seg"]
                ))
                print(f"Body ids via ins_seg: {body_ids}")
                print(f"Body ids via scene: {env.simulator.scene.objects_by_id.keys()}")

                plt.show()

            # if done:
            #     break

    env.close()

if __name__ == "__main__":
    main()

[mjlbach]

Note, I updated the issue title, but it could be the case that the instance ids themselves are wrong rather than the conversion of instance id to pybullet id. I can't test with/without the optimized renderer because the optimized renderer does not work on macos. I can try on linux I guess with the non-optimized renderer.

[mjlbach]

I've also noticed that it's not returning the correct number of pybullet ids... Ins seg ids: [ 8 10 112 244 452 638 640 724 738] Body ids via ins_seg: [ -1 8 10 112 244] Body ids via scene: dict_keys([2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 54, 55, 56, 57, 58, 60, 61, 74, 78, 79, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95])