Table not segmented in PutCarrotOnPlateInScene-v1 environment

[ozgraslan]

Hello, Thank you for this great work!

I am using "PutCarrotOnPlateInScene-v1" environment, and I want to extract the segmentation map of the environment which I will use in a learning task.

However, the table is not segmented. I checked the human rendering mode, and there is a table object but it is not seperated from the environment.

Is there a way to fix this?

Example code:

import gymnasium as gym
import numpy as np
import matplotlib.pyplot as plt

from mani_skill.envs.tasks.digital_twins.bridge_dataset_eval import *
from mani_skill.envs.sapien_env import BaseEnv
from mani_skill.sensors.camera import camera_observations_to_images

env: BaseEnv = gym.make(
  "PutCarrotOnPlateInScene-v1",
  obs_mode="rgb+segmentation",
  render_mode="human",
  num_envs=1, # if num_envs > 1, GPU simulation backend is used.
)
obs, _ = env.reset()
images = camera_observations_to_images(obs["sensor_data"]["3rd_view_camera"])
rgb = np.concatenate(images["rgb"].cpu().numpy(), axis=1)
segment = np.concatenate(images["segmentation"].cpu().numpy(), axis=1)

combined = np.concatenate([rgb, segment], axis=0)
plt.imshow(combined)
plt.show()

Output:

Human rendering mode:

[StoneT2000]

yes unfortunately the original environment this one is based on (The https://github.com/simpler-env/SimplerEnv project) uses a little bit of a strange table which is merged together with the background walls and floors. As a result the segmentation won't make sense really. In the future we have ongoing work expanding on SimplerEnv to make a larger set of cleaner real2sim evaluation envs.

Regardless one fix you can do for now is to manually annotate the table. The table is fixed in this environment all the time

[ozgraslan]

Thank you for your response!

I solved this by separating the table and the scene using blender, and adding them as separate objects.

[StoneT2000]

that also works!