Closed samibouziri closed 7 months ago
samibouziri
commented
7 months ago FYI this is what you get for the default plane cfg:
and this is what you get when placing the pov camera at the same at the same position and orientation as the raycaster camera for the rough terrain like in the tutorial (expected behavious a red rectangle):
as you can see we get the expected behavior (more or less. it is shifted a bit as you can see in the second photo) for positive x,y
pascal-roth
commented
7 months ago ./orbit.sh -p source/standalone/tutorials/04_sensors/run_ray_caster_camera.py The outputted points are not always on the rough terrain.
The rough_plane.usd of Nuclues has a translation of (-40, -40, 0), if you remove it, the output fits. In our minimal example, we tested with our local version that does not have such an offset. It is not a fault of the raycaster. Will adapt the version so that it fits to the Nucleus asset.
For a fix insert the following code at line 88:
# -- Set origin of rough terrain to (0, 0, 0)
ground_prim = prim_utils.get_prim_at_path("/World/ground/mesh")
ground_prim.GetAttribute("xformOp:translate").Set((0.0, 0.0, 0.0))When replacing the terrain in the mesh prim path with a shape config we get an error. When replacing the terrain in the mesh prim path with with the table that we have in other tutorials we get an error. When replacing the terrain in the mesh prim path with "{ISAAC_NUCLEUS_DIR}/Props/YCB/Axis_Aligned_Physics/003_cracker_box.usd", there are no errors but the point cloud is under the terrain and none of the points are on the box.
I am having issues understanding what you did here precisely. The mesh_prim_path in this example only has to be set in the RayCasterCameraCfg. This is defined as follows:
mesh_prim_paths: list[str] = MISSING
"""The list of mesh primitive paths to ray cast against.
Note:
Currently, only a single static mesh is supported. We are working on supporting multiple
static meshes and dynamic meshes.
"""which means it has to be a prim_path that is already included in the scene. Neither a config nor a table that is not important yet can be used. Also the some_path.usd cannot be used as the object needs to be first added to the scene and then you can insert the corresponding prim path
samibouziri
commented
7 months ago @pascal-roth Thanks for the quick answer. So here is the full modified code to cast the point cloud on the box
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
"""
This script shows how to use the ray-cast camera sensor from the Orbit framework.
The camera sensor is based on using Warp kernels which do ray-casting against static meshes.
.. code-block:: bash
# Usage
./orbit.sh -p source/standalone/tutorials/04_sensors/run_ray_caster_camera.py
"""
"""Launch Isaac Sim Simulator first."""
import argparse
from omni.isaac.orbit.app import AppLauncher
# add argparse arguments
parser = argparse.ArgumentParser(description="This script demonstrates how to use the ray-cast camera sensor.")
parser.add_argument("--num_envs", type=int, default=16, help="Number of environments to generate.")
parser.add_argument("--save", action="store_true", default=False, help="Save the obtained data to disk.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# parse the arguments
args_cli = parser.parse_args()
# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app
"""Rest everything follows."""
import os
import torch
import traceback
import carb
import omni.isaac.core.utils.prims as prim_utils
import omni.replicator.core as rep
import omni.isaac.orbit.sim as sim_utils
from omni.isaac.orbit.sensors.ray_caster import RayCasterCamera, RayCasterCameraCfg, patterns
from omni.isaac.orbit.utils import convert_dict_to_backend
from omni.isaac.orbit.utils.assets import ISAAC_NUCLEUS_DIR
from omni.isaac.orbit.utils.math import project_points, unproject_depth
def define_sensor() -> RayCasterCamera:
"""Defines the ray-cast camera sensor to add to the scene."""
# Camera base frames
# In contras to the USD camera, we associate the sensor to the prims at these locations.
# This means that parent prim of the sensor is the prim at this location.
prim_utils.create_prim("/World/Origin_00/CameraSensor", "Xform")
# prim_utils.create_prim("/World/Origin_01/CameraSensor", "Xform")
# Setup camera sensor
camera_cfg = RayCasterCameraCfg(
prim_path="/World/Origin_.*/CameraSensor",
mesh_prim_paths=["/World/Objects/Box"],
update_period=0.1,
offset=RayCasterCameraCfg.OffsetCfg(pos=(0.0, 0.0, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
data_types=["distance_to_image_plane", "normals", "distance_to_camera"],
debug_vis=True,
pattern_cfg=patterns.PinholeCameraPatternCfg(
focal_length=24.0,
horizontal_aperture=20.955,
height=480,
width=640,
),
)
# Create camera
camera = RayCasterCamera(cfg=camera_cfg)
return camera
def design_scene():
# Populate scene
# -- terrain
cfg_ground = sim_utils.GroundPlaneCfg()
cfg_ground.func("/World/ground", cfg_ground)
# -- Lights
cfg = sim_utils.DistantLightCfg(intensity=600.0, color=(0.75, 0.75, 0.75))
cfg.func("/World/Light", cfg)
# -- object
cfg = sim_utils.UsdFileCfg(usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/YCB/Axis_Aligned_Physics/003_cracker_box.usd")
cfg.func("/World/Objects/Box", cfg, translation=(0.0, 0.0, 0.02))
# -- Sensors
camera = define_sensor()
# return the scene information
scene_entities = {"camera": camera}
return scene_entities
def run_simulator(sim: sim_utils.SimulationContext, scene_entities: dict):
"""Run the simulator."""
# extract entities for simplified notation
camera: RayCasterCamera = scene_entities["camera"]
# Create replicator writer
output_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "output", "ray_caster_camera")
rep_writer = rep.BasicWriter(output_dir=output_dir, frame_padding=3)
# Set pose: There are two ways to set the pose of the camera.
# -- Option-1: Set pose using view
eyes = torch.tensor([[2.5, 2.5, 2.5]], device=sim.device)
targets = torch.tensor([[0.0, 0.0, 0.0]], device=sim.device)
camera.set_world_poses_from_view(eyes, targets)
# -- Option-2: Set pose using ROS
# position = torch.tensor([[2.5, 2.5, 2.5]], device=sim.device)
# orientation = torch.tensor([[-0.17591989, 0.33985114, 0.82047325, -0.42470819]], device=sim.device)
# camera.set_world_poses(position, orientation, indices=[0], convention="ros")
# Simulate physics
while simulation_app.is_running():
# Step simulation
sim.step()
# Update camera data
camera.update(dt=sim.get_physics_dt())
# Print camera info
print(camera)
print("Received shape of depth image: ", camera.data.output["distance_to_image_plane"].shape)
print("-------------------------------")
# Extract camera data
if args_cli.save:
# Extract camera data
camera_index = 0
# note: BasicWriter only supports saving data in numpy format, so we need to convert the data to numpy.
if sim.backend == "torch":
# tensordict allows easy indexing of tensors in the dictionary
single_cam_data = convert_dict_to_backend(camera.data.output[camera_index], backend="numpy")
else:
# for numpy, we need to manually index the data
single_cam_data = dict()
for key, value in camera.data.output.items():
single_cam_data[key] = value[camera_index]
# Extract the other information
single_cam_info = camera.data.info[camera_index]
# Pack data back into replicator format to save them using its writer
rep_output = dict()
for key, data, info in zip(single_cam_data.keys(), single_cam_data.values(), single_cam_info.values()):
if info is not None:
rep_output[key] = {"data": data, "info": info}
else:
rep_output[key] = data
# Save images
rep_output["trigger_outputs"] = {"on_time": camera.frame[camera_index]}
rep_writer.write(rep_output)
# Pointcloud in world frame
points_3d_cam = unproject_depth(
camera.data.output["distance_to_image_plane"], camera.data.intrinsic_matrices
)
# Check methods are valid
im_height, im_width = camera.image_shape
# -- project points to (u, v, d)
reproj_points = project_points(points_3d_cam, camera.data.intrinsic_matrices)
reproj_depths = reproj_points[..., -1].view(-1, im_width, im_height).transpose_(1, 2)
sim_depths = camera.data.output["distance_to_image_plane"].squeeze(-1)
torch.testing.assert_close(reproj_depths, sim_depths)
def main():
"""Main function."""
# Load kit helper
sim = sim_utils.SimulationContext()
# Set main camera
sim.set_camera_view([2.5, 2.5, 2.5], [0.0, 0.0, 0.0])
# design the scene
scene_entities = design_scene()
# Play simulator
sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run simulator
run_simulator(sim=sim, scene_entities=scene_entities)
if __name__ == "__main__":
try:
# run the main execution
main()
except Exception as err:
carb.log_error(err)
carb.log_error(traceback.format_exc())
raise
finally:
# close sim app
simulation_app.close()and here is the output above the ground:
and this is the output under the ground:
samibouziri
commented
7 months ago @pascal-roth Thanks again for your answer. From the comment above, I understand now what's happening here. Basically same issue as the ground but instead of having some translation issues we have a scaling issue. It was good to know that the ray caster works, just if you can add in the documentation that the ray caster does not take into account any alteration on the mesh (scale, translation, rotation, ...) that would be amazing. It can also be a feature to add in the future :D .
pascal-roth
commented
7 months ago No worries. These things are actually already in a PR and should be merged soon.
gongceerhao
commented
1 week ago gongceerhao
commented
1 week ago What happen？
samibouziri
commented
1 week ago Well, if nothing changed from my last post, you can only raycast on one static asset. And the raycaster only considers the mesh of that asset in it s default state. For boxes the default state should be a box centered around (0,0,0). So no matter where you translate the box, if it is not done in the creation of that mesh (which cannot be done using isaac lab functionalities; in isaac lab you first create the asset around 0,0,0 and then it is translated during the initialization of the sim), the raycaster will not perceived that translation. To percieve the translation you should first create the Mesh using another software and translate it there and then in isaac lab create an asset from that Mesh.
Describe the bug
The Ray Caster Camera does not work properly. It creates hitting points in the wrong places.
Steps to reproduce
A way to reproduce the bug is to run the Ray Caster Camera tutorial with one camera: ./orbit.sh -p source/standalone/tutorials/04_sensors/run_ray_caster_camera.py The outputted points are not always on the rough terrain. More over :
Note: I have also tried to draw the point cloud myself following the procedure given in the documentation and had the same result.
System Info
Describe the characteristic of your environment:
Checklist