Inquiring About Integration Method for NVIDIA ISAAC ROS's isaac_ros_visual_slam with rapyuta-robotics/rclUE

[Buddies-as-you-know]

Overview

• I am interested in integrating the NVIDIA ISAAC ROS's isaac_ros_visual_slam component into the rclUE package, which connects Unreal Engine 5 with ROS2.
• isaac_ros_visual_slam provides SLAM (Simultaneous Localization and Mapping) capabilities, and I am particularly interested in its application within the real-time 3D environment of Unreal Engine.

Objective

• My goal is to implement SLAM functionalities within the Unreal Engine simulation environment to achieve advanced robot navigation and environmental perception.
• By integrating isaac_ros_visual_slam, I aim to seamlessly combine robot operations in Unreal Engine with ROS2 functionalities.

Questions

1. What is the best approach to integrate isaac_ros_visual_slam into rclUE?
2. Are there any specific dependencies or configurations that I should be aware of?
3. How might isaac_ros_visual_slam impact data exchange between Unreal Engine and ROS2?

Additional Information

• Please provide details of your current development environment, including the versions of Unreal Engine 5 and ROS2 you are using.
• Insights from other users or developers who have experience with isaac_ros_visual_slam would be highly appreciated.

[yuokamoto]

I'm not very familliar with isaac_ros_visual_slam but

it seems isaac_ros_visual_slam uses stereo camera and IMU. Since you can use camera component to publish image topic https://rapyutasimulationplugins.readthedocs.io/en/devel/doxygen_generated/html/d9/d91/class_u_r_r_r_o_s2_camera_component.html Currently RapyutaSimulationPlugins doesn;t have IMU sensor, we need to add it.

rclUE/RapyutaSImulationPlugins are mainly tested in Ubuntu20.04/22.04 and ROS 2 foxy/humble

[Buddies-as-you-know]

Is it possible to integrate not only isaac_ros_visual_slam but also vision's slam? I want to autopilot turtlebot from images. I want to use slam from a monocular or depth image in UE to perform auto navigation with nav2, but I don't know how.

[yuokamoto]

You can use depth camera by setting URRROS2CameraComponent::CameraType = DEPTH CameraType = EROS2CameraType::RGB

[Buddies-as-you-know]

I want to use Rtabmap with a depth camera.

• How should I configure the depth camera settings on the Unreal Engine side?
• How can I pass this depth camera footage to Rtabmap?
• I would like to perform this tutorial. How should I proceed?

[yuokamoto]

• you can attach URRROS2CameraComponent and change type to CameraType = DEPTH
• since camera image is published as rostopic, you can change topic name to make it same as Rtabmap setting
• you need to 1) specify necessary ROS interface for Rtabmap, 2) impl/config UE to have that.

[Buddies-as-you-know]

Would it be advisable to set this up on the Unreal Engine side? Encoding: RGB ->depth

[Buddies-as-you-know]

ros2 launch rtabmap_demos turtlebot3_scan.launch.py Is the rtabmap running on the depth camera?

[yuokamoto]

Here is camera type setting