Open ori-gadot opened 1 month ago
I can confirm that this issue already happens by the time that the image is read by grab_images in the observation group, but only for Tiled Cameras (standard USD cameras look like they work).
I've combined your repro steps in a single standalone file for easier troubleshooting here: https://gist.github.com/garylvov/d9cd169883838bd5b43d2f7b6d75c52c
@Dhoeller19 @Mayankm96 I believe that this may be on Isaac sim side, or maybe the buffers in tiled_camera.py
aren't created/used correctly
same issue.
Thanks for the report, I don't believe we can currently have multiple tiled cameras at different resolutions as the resolution settings are global. We'll look into adding support for this.
@kellyguo11 I believe this issue still happens even if the multiple tiled cameras are the same resolution
Yes, you are right. It seems currently we can only have one tiled camera. The fix is in the works to support multiple tiled cameras, including at different resolutions. Hopefully it'll be made available in the next release.
@kellyguo11 Just to add to the discussion, as described in https://github.com/isaac-sim/IsaacLab/issues/992 I can successfully render 2x tiled cameras and get different images (same resolution), but there is an issue with correctly setting both poses.
@elle-miller Has an interesting observation @kellyguo11 with #992 .
EDIT: I believe that #1070 is still an issue as #992 looks like it produces different images, but I believe that they are the same image just with different noise levels.
I can confirm that when I set the camera views with set_world_poses_from_view
, I am able to get different images from different tiled cameras in the same environment.
Using set_world_poses_from_view : https://gist.github.com/glvov-bdai/767ad9d76ea2a10b625fe7cbfbc21c20
Hi, I have a similar issue.
When I have a Franka environment with a front-facing and wrist camera, I can get the Tiled Images to render properly. However, the moment I have two environments, the TiledCamera on the wrist has an incorrect orientation:
Above is what my wrist camera is supposed to look like. It tracks the arm normally.
This is what the wrist camera looks like if I have multiple environments. However, my front camera renders properly (i.e., it doesn't seem like the wrist camera overwrites the front camera. I will sat that the front camera suffers from #1031
Normal front camera view
Front Camera view if I have parallelized environments. Here, the block appear distorted.
@kensukenk Can you please be a little bit more explicit about what the problem is? Is it the camera not being in the desired orientation when using Tiled Cameras but it is in the correct orientation with USD Cameras?
If you are using set_world_poses for your camera (or anything related to pose), I'd double check that the pose convention is consistent ("opengl" or "ros" or "world) for all cameras. From what I was able to piece together from your photos, I'd guess that the usd camera is in OpenGL format, whetheras the Tiled camera may be in ROS format, but I'm not sure.
from omni.isaac.lab.sensors.camera.camera.py
Since different fields use different conventions for camera orientations, the method allows users to
set the camera poses in the specified convention. Possible conventions are:
- :obj:`"opengl"` - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
- :obj:`"ros"` - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
- :obj:`"world"` - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention
See :meth:`omni.isaac.lab.sensors.camera.utils.convert_orientation_convention` for more details
on the conventions.
Can you please share code to replicate your results? I'm surprised that the multiple cameras don't override each other and would be curious to investigate this further.
Sure, my code is a modification of the Franka Cube Lift env. My issue is that when I use only one environment, my TiledCamera renders correctly. However, when I use two or more environments in parallel (and change nothing else), the TiledCamera on the franka wrist flips orientation.
I added cameras to the lift_env_cfg.py file as
'class ObjectTableSceneCfg(InteractiveSceneCfg): """Configuration for the lift scene with a robot and a object. This is the abstract base implementation, the exact scene is defined in the derived classes which need to set the target object, robot and end-effector frames """
# robots: will be populated by agent env cfg
robot: ArticulationCfg = MISSING
# end-effector sensor: will be populated by agent env cfg
ee_frame: FrameTransformerCfg = MISSING
# target object: will be populated by agent env cfg
object: RigidObjectCfg = MISSING
#camera: TiledCameraCfg = MISSING
wrist_cam: CameraCfg | TiledCameraCfg = MISSING
front_cam: CameraCfg | TiledCameraCfg = MISSING'
I also set self.rerender_on_reset=True, in case that matters.
and ` @configclass class FrankaCubeLiftEnvCfg(LiftEnvCfg): def __post_init__(self):
super().__post_init__()
# Set Franka as robot
self.scene.robot = FRANKA_PANDA_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
# Set actions for the specific robot type (franka)
self.actions.body_joint_pos = mdp.JointPositionActionCfg(
asset_name="robot", joint_names=["panda_joint.*"], scale=0.5, use_default_offset=True
)
self.actions.finger_joint_pos = mdp.BinaryJointPositionActionCfg(
asset_name="robot",
joint_names=["panda_finger.*"],
open_command_expr={"panda_finger_.*": 0.04},
close_command_expr={"panda_finger_.*": 0.0},
)
# Set the body name for the end effector
#self.commands.object_pose.body_name = "panda_hand"
'''# Set Cube as object
self.scene.object = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Object",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.5, 0, 0.055], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/DexCube/dex_cube_instanceable.usd",
scale=(0.8, 0.8, 4.0),
rigid_props=RigidBodyPropertiesCfg(
solver_position_iteration_count=16,
solver_velocity_iteration_count=1,
max_angular_velocity=1000.0,
max_linear_velocity=1000.0,
max_depenetration_velocity=5.0,
disable_gravity=False,
),
),
)'''
self.scene.left = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/left",
spawn=sim_utils.CuboidCfg(
size=[0.05, 0.05, 0.2],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 0.0, 0.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0.6, 0.07, 0.18), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.object = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Object",
spawn=sim_utils.CuboidCfg(
size=[0.05, 0.05, 0.2],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(0.0, 1.0, 0.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.6, 0, 0.18], rot=[1.0, 0.0, 0.0, 0.0]),
)
self.scene.right = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/right",
spawn=sim_utils.CuboidCfg(
size=[0.05, 0.05, 0.2],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 0.0, 0.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0.6, -0.07, 0.18), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.base = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/base",
spawn=sim_utils.CuboidCfg(
size=[0.7, 0.7, 0.08],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=10.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(0.0, 0.0, 1.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0.5, -0.00, 0.05), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.wall1 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/wall1",
spawn=sim_utils.CuboidCfg(
size=[2.5, 0.1, 3.],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=10.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 1.0, 1.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0., 1.50, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.wall2 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/wall2",
spawn=sim_utils.CuboidCfg(
size=[2.5, 0.3, 3.],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=10.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 1.0, 1.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0., -1.50, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.wall3 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/wall3",
spawn=sim_utils.CuboidCfg(
size=[0.3, 3, 3.],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=10.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 1.0, 1.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(1.4, 0.00, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
)
self.scene.wall4 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/wall4",
spawn=sim_utils.CuboidCfg(
size=[0.3, 3, 3.],
rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=10.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 1.0, 1.0), metallic=0.2),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(-1.4, 0.00, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
)
# Listens to the required transforms
marker_cfg = FRAME_MARKER_CFG.copy()
marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
marker_cfg.prim_path = "/Visuals/FrameTransformer"
self.scene.ee_frame = FrameTransformerCfg(
prim_path="{ENV_REGEX_NS}/Robot/panda_link0",
debug_vis=False,
visualizer_cfg=marker_cfg,
target_frames=[
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/panda_hand",
name="end_effector",
offset=OffsetCfg(
pos=[0.0, 0.0, 0.1034],
),
),
],
)
self.scene.wrist_cam = TiledCameraCfg(
prim_path="{ENV_REGEX_NS}/Robot/panda_hand/wrist_cam",
update_period=0.01,
height=128,
width=128,
data_types=["rgb"],
spawn=PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20)
),
offset=TiledCameraCfg.OffsetCfg(pos=(0.1, 0.0, 0.0), rot=(0.6848517, -0.1750107, -0.1748713, 0.6853973), convention="ros"),
)
self.scene.front_cam = TiledCameraCfg(
prim_path="{ENV_REGEX_NS}/front_cam",
update_period=0.01,
height=128,
width=128,
data_types=["rgb"],
spawn=PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20)
),
offset=TiledCameraCfg.OffsetCfg(pos=(0.75, 0.0, 0.75), rot=(-0.1259888, 0.6960601, 0.695506, -0.1260892 ), convention="ros"),
)
`
In particular, I defined the TiledCameras in the last text chunks directly above (self.scene.front_cam, self.scene.wrist_cam)
EDIT: @kensukenk thank you for submitting the report at https://github.com/isaac-sim/IsaacLab/issues/1127 !
@kensukenk Thanks for the additional information. This looks like a separate issue to me to do with how poses are being set, do you mind opening a new issue with this information so we can keep this issue more focused to cameras overriding each other?
Also if you're able to provide the steps to reproduce as a standalone file in the style of this gist: https://gist.github.com/garylvov/d9cd169883838bd5b43d2f7b6d75c52c
that would help the dev team trouble shoot this issue faster. Otherwise, we have to recreate/keep track of your modifications, which can slow things down.
Thanks!
Bug Description
When using two tiled cameras in the scene, the last camera overrides all observations, resulting in both cameras capturing the same image.
Steps to reproduce
CartpoleRGBCameraEnvCfg
environment with two environments and two tiled cameras.CartpoleRGBCameraSceneCfg
class in the following file:source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_camera_env_cfg.py
, adding two tiled cameras as shown below:RGBObservationsCfg
to include observations for both cameras:source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_env_cfg.py
for clearer image results:Additional context
Two identical tiled cameras with slightly different orientations (one capturing more of the ground, the other more of the sky) were used. Observation data was saved as images for inspection.
Running the following scenarios, only the first one reproduces the bug:
Scenario 1: Running two tiled cameras in two environments causes the bug—both image files show the second camera’s view.
Camera 1
Camera 2
Scenario 2: Running two tiled cameras in a single environment works correctly—each camera renders its own FOV.
Camera 1
Camera 2
Scenario 3: Running two non-tiled cameras in two environments works as expected.
Camera 1
Camera 2
Note: Although it may be unrelated, there is a noticeable decline in image quality in the first scenario compared to the other two.