[Question] Will more sensors be supported in the future?

[shaoxiang]

Question

Thank you very much for providing such an excellent robot simulation and learning platform. The robot sensors I am using in simulation are LiDAR and IMU. Previously, I used RTX lidar in isaac sim, which supports Ouster, Velodyne, etc. I need to migrate them to isaac lab now. I noticed that Velodyne lidar has been added #616 , but it is implemented using RayCaster. The information contained in RayCasterData is far inferior to RTX Lidar. For example, without intensity information. (As mentioned @Mayankm96 in #532 I think it is better to move the implementation to use RTX-based sensor since that will capture the LiDAR noise better. ) So, how to migrate RTX Lidar to isaac lab, and is there any official support plan? (I am referring to this open-source work: go2_omniverse

[jakethesnake420]

I am working on the RTX lidar support privately right now and im making good progress. It will work just like the camera interface. Ill make a Pr when its some what usable.

    lidar = RTXRayCasterCfg(
        prim_path="{ENV_REGEX_NS}/Robot/base/lidar",
        offset=RTXRayCasterCfg.OffsetCfg(pos=(0.0, 0.0, 20.0)),
        spawn=sim_utils.LidarCfg()
    )

[shaoxiang]

Very good work, looking forward to your further updates, this is enough for many perceptions. But I am afraid this is not a real RTX lidar, I understand that this cannot obtain lidar intensity information. @jakethesnake420

[jakethesnake420]

@shaoxiang I am pretty sure intensity information is there. You can find more information here https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/isaac_sim_sensors_rtx_based_lidar/annotator_descriptions.html

You can test out my PR now. https://github.com/isaac-sim/IsaacLab/pull/802

[shaoxiang]

Thank you very much for your excellent work. Through your code I successfully obtained the intensity information of the LiDAR printing (f "Intensity for {key}: {lidar _data. intensity}") log shows: Intensity for /World/envs/env_0/Robot/body/Lidar: tensor([0.0307, 0.0135, 0.0092, ..., 0.0063, 0.0167, 0.0148]) But when I found out that switching LidarType from VELODYNE_VLS128 to HESAI_PandarXT_32 would result in this error and unable to obtain radar data, do you know how to solve it?

2024-08-09 08:43:36 [152,209ms] [Warning] [omni.timeline.plugin] Deprecated: direct use of ITimeline callbacks is deprecated. Use ITimeline::getTimeline (Python: omni.timeline.get_timeline_interface) instead. 2024-08-09 08:45:09 [245,662ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] getProfileJsonAtPaths could not find config file: "Hesai_XT32_SD10", in extension or in supplied paths: 2024-08-09 08:45:09 [245,662ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/extsbuild/omni.sensors.nv.common/data/lidar/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,663ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/HESAI/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,663ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/NVIDIA/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,663ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/Ouster/OS0/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,664ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/Ouster/OS1/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,664ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/Ouster/OS2/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,664ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/SICK/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,664ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/SLAMTEC/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,665ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/Velodyne/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,665ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/ZVISION/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,665ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] D:/omniverse/pkg/isaac-lab/mylab/IsaacLab/source/exts/omni.isaac.sensor/data/lidar_configs/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,666ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] d:/omniverse/pkg/isaac-lab/mylab/isaaclab/_isaac_sim/extscache/omni.sensors.nv.common-1.2.2-isaac+wx64.r.cp310/data/lidar/Hesai_XT32_SD10.json 2024-08-09 08:45:09 [245,666ms] [Error] [omni.sensors.nv.profile_reader.ext.plugin] getProfileJsonAtPaths is creating a minimal modality specific profile 2024-08-09 08:45:09 [245,666ms] [Warning] [lidar.lidarprofilereader] Member rotationDirection not found! 2024-08-09 08:45:09 [245,666ms] [Warning] [omni.sensors.nv.profile_reader.ext.plugin] This style of emitters will be deprecated in the near future. Use emitterStates approach instead. 2024-08-09 08:45:09 [245,952ms] [Warning] [rtx.rtxsensor.plugin] Using User-specified coordinate frame quaternion (0.000000, 0.000000, 0.000000, 1.000000) in RTXSensor