Closed titoirfan closed 2 months ago
We encountered similar issues with the TiledCamera
. For us, the CartpoleRGBCamera
returns complete black images only. In a second test, a TiledCamera
sensor was added to the FrankaCubeLift
environment. In there, we get noisy images that don't correspond to the scene in any way. When the camera is swapped to the standard non-tiled version, everything works fine.
System Info:
Setup1:
Setup2:
The target system for our experiments would be an HPC with L40S and H100 GPUs
Code:
Test1 CartpoleRGBCamera
:
import argparse
from omni.isaac.lab.app import AppLauncher
parser = argparse.ArgumentParser(description="TEST")
parser.add_argument("--num_envs", type=int, default=2, help="")
AppLauncher.add_app_launcher_args(parser)
args_cli = parser.parse_args()
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app
import torch
import omni.replicator.core as rep
from omni.isaac.lab.utils import convert_dict_to_backend
from omni.isaac.lab_tasks.direct.cartpole.cartpole_camera_env import CartpoleRGBCameraEnvCfg, CartpoleCameraEnv
env_cfg = CartpoleRGBCameraEnvCfg()
env_cfg.scene.num_envs = args_cli.num_envs
env_cfg.write_image_to_file = True
env = CartpoleCameraEnv(cfg=env_cfg, render_mode="rgb_array")
rep_writer = rep.BasicWriter(
output_dir="/workspace/isaaclab/camera_output",
frame_padding=3
)
for i in range(100):
a = torch.rand(env.num_envs, env.num_actions, device=env.device)
observation, reward, terminated, truncated, info = env.step(a)
camera = env.scene["tiled_camera"]
for camera_index in range(camera.data.output.shape[0]):
single_cam_data = convert_dict_to_backend(camera.data.output[camera_index], backend="numpy")
rep_output = {"annotators": {}}
for key, data in zip(single_cam_data.keys(), single_cam_data.values()):
if data.shape[-1] == 4:
data = data[..., :-1]
rep_output["annotators"][key] = {"render_product": {"data": data}}
rep_output["trigger_outputs"] = {"on_time": camera.frame[camera_index]}
rep_writer.write(rep_output)
Tiled camera output:
Test2 FrankaCubeLift
:
import argparse
from omni.isaac.lab.app import AppLauncher
parser = argparse.ArgumentParser(description="TEST")
parser.add_argument("--num_envs", type=int, default=2, help="")
AppLauncher.add_app_launcher_args(parser)
args_cli = parser.parse_args()
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app
import torch
import omni.replicator.core as rep
import omni.isaac.lab.sim as sim_utils
from omni.isaac.lab.sensors import CameraCfg, TiledCameraCfg
from omni.isaac.lab.managers import ObservationTermCfg as ObsTerm
from omni.isaac.lab.managers import SceneEntityCfg
from omni.isaac.lab_tasks.manager_based.manipulation.lift.config.franka.joint_pos_env_cfg import FrankaCubeLiftEnvCfg
from omni.isaac.lab.envs import ManagerBasedRLEnv
from omni.isaac.lab.utils import convert_dict_to_backend
def camera_img(
env: ManagerBasedRLEnv,
cam_cfg: SceneEntityCfg = SceneEntityCfg("camera"),
) -> torch.Tensor:
camera_output = env.scene[cam_cfg.name].data.output["rgb"]
camera_output = camera_output[..., :-1] # cut depth chanel
return camera_output.flatten(start_dim=1)
class FrankaObjectLiftEnvCfg(FrankaCubeLiftEnvCfg):
def __post_init__(self):
super().__post_init__()
self.decimation = 6
self.episode_length_s = 2.5
self.sim.dt = 0.01
self.scene.camera = CameraCfg(
prim_path="{ENV_REGEX_NS}/Table/camera_sensor",
update_period=self.decimation * self.sim.dt,
height=84,
width=84,
data_types=["rgb"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
),
offset=CameraCfg.OffsetCfg(pos=(0.0, -0.4, 0.4),
rot=(0.6532814862382034, -0.2705980584036595,
0.2705980656543066, 0.6532814687335938),
convention="world"),
)
self.scene.camera = TiledCameraCfg(
prim_path="{ENV_REGEX_NS}/Table/camera_sensor",
update_period=self.decimation * self.sim.dt,
offset=TiledCameraCfg.OffsetCfg(pos=(0.0, -0.4, 0.4),
rot=(0.6532814862382034, -0.2705980584036595,
0.2705980656543066, 0.6532814687335938),
convention="world"),
data_types=["rgb"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
),
width=84,
height=84,
)
self.observations.policy.image = ObsTerm(func=camera_img)
env_cfg = FrankaObjectLiftEnvCfg()
env_cfg.scene.num_envs = args_cli.num_envs
env = ManagerBasedRLEnv(cfg=env_cfg, render_mode="rgb_array")
rep_writer = rep.BasicWriter(
output_dir="/workspace/isaaclab/camera_output",
frame_padding=3
)
for i in range(100):
a = torch.randn_like(env.action_manager.action)
observation, reward, terminated, truncated, info = env.step(a)
camera = env.scene["camera"]
for camera_index in range(camera.data.output.shape[0]):
single_cam_data = convert_dict_to_backend(camera.data.output[camera_index], backend="numpy")
rep_output = {"annotators": {}}
for key, data in zip(single_cam_data.keys(), single_cam_data.values()):
if data.shape[-1] == 4:
data = data[..., :-1]
rep_output["annotators"][key] = {"render_product": {"data": data}}
rep_output["trigger_outputs"] = {"on_time": camera.frame[camera_index]}
rep_writer.write(rep_output)
Console command: ./isaaclab.sh -p test.py --headless --enable_cameras --num_envs 2
Tiled camera output:
Standard camera output:
Thanks @ArneKlages4444 for sharing and confirming the issue!
Could the developers please take another look at this issue?
The tiled camera output is expected to be different than what we get from the renderer. Currently, the RGB tiled rendering API only provides ambient RGB, which does not capture any lighting and shadow information. We are working on a full tiled rendering API that will provide the same data as what we would achieve with the renderer.
Thank you for the clarification, looking forward to the full tiled rendering API.
Question
Hi there,
As I was migrating my code from Orbit to IsaacLab, I tried to change my RGB
Camera
sensors into RGBTiledCamera
sensors. However, as I was testing theTiledCamera
sensor, I noticed that it produces very different images compared toCamera
.I tried to add an RGB
TiledCamera
sensor to the filesource/standalone/tutorials/04_sensors/add_sensors_on_robot.py
, then visualize the images rendered by both the RGBTiledCamera
and RGBCamera
usingmatplotlib.pyplot
.Aside from adding
matplotlib.pyplot as plt
andTiledCameraCfg
to the imports, I added these lines toSensorsSceneCfg
and these to the end of
run_simulator()
.I then just invoke
python source/standalone/tutorials/04_sensors/add_sensors_on_robot.py --enable_cameras
.Image produced by![Camera Image](https://github.com/isaac-sim/IsaacLab/assets/42381907/d7ed7967-6895-4a99-b3b7-d2995522946f)
Camera
:Image produced by![TiledCamera Image](https://github.com/isaac-sim/IsaacLab/assets/42381907/9748115f-730d-492c-90c6-acc7abe6e9cb)
TiledCamera
:By doing some similar steps, I also compared the images produced by both classes on
CartpoleRGBCameraEnvCfg
.Image produced by![Camera Image](https://github.com/isaac-sim/IsaacLab/assets/42381907/60a561a8-1cfc-4031-9d87-768873de099d)
Camera
:Image produced by![TiledCamera Image](https://github.com/isaac-sim/IsaacLab/assets/42381907/af5a8fa1-52c3-4f7c-9944-721f24fd598a)
TiledCamera
:I am not sure if this is a bug, as in the Cartpole case the images produced by
TiledCamera
might be more desirable. However, in other cases (e.g. quadruped locomotion), I don't think that the images produced byTiledCamera
are usable as they don't reflect how a camera behaves in the real world.I tried to read the docs but all I get from it is that
TiledCamera
is supposed to be a faster and more efficientCamera
for multi-camera simulation purposes.Hence, my questions are:
TiledCamera
andCamera
intended?TiledCamera
produce images similar toCamera
?Thank you in advance!
Edit: Here is my setup, I use the conda environment provided by IsaacLab