search

NVIDIA-ISAAC-ROS / isaac_ros_depth_segmentation

NVIDIA-accelerated, deep learned depth segmentation and obstacle field ranging using Bi3D
Apache License 2.0
45 stars 8 forks source link

Proximity Segmentation in Isaac Gym (the Omniverse version) #9

Closed joehays closed 1 year ago

joehays commented 1 year ago

Hello,

Is there any support to be able to use the Proximity Segmentation capabilities from Isaac ROS Proximity Segmentation in parallel environments in Omniverse Sim/Gym? The ROS2 layer would have to be stripped away as it's not appropriate for the parallel Gym environments used during the RL training. Is this possible?

hemalshahNV commented 1 year ago

Possible, yes, but it would take some work to strip away the ROS 2 layer and integrate the same underlying libraries with Isaac Gym. Alternatively, you can integrate Isaac Gym and Isaac ROS through your own ROS 2 application.

joehays commented 1 year ago

ROS is inappropriate when I have over 10k parallel environments running in Isaac Gym. The power of Isaac Gym is only fully realized when all code runs on the GPU. Having ROS in the middle removes the primary performance benefits of Isaac Gym.

On Mon, Mar 6, 2023 at 1:59 PM Hemal Shah @.***> wrote:

Possible, yes, but it would take some work to strip away the ROS 2 layer and integrate the same underlying libraries with Isaac Gym. Alternatively, you can integrate Isaac Gym and Isaac ROS through your own ROS 2 application.

— Reply to this email directly, view it on GitHub https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_proximity_segmentation/issues/9#issuecomment-1456773588, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAYEB4O2RAB4C4GNWBPS5DW2YXXHANCNFSM6AAAAAAVPCEYJM . You are receiving this because you authored the thread.Message ID: <NVIDIA-ISAAC-ROS/isaac_ros_proximity_segmentation/issues/9/1456773588@ github.com>

jaiveersinghNV commented 1 year ago

Isaac ROS Proximity Segmentation functions using the underlying Bi3D model, which is independently available on NGC here. The model is distributed as a set of .onnx files, which can be loaded by any standard inference engine (Triton, TensorRT, Pytorch, etc.).

You should be able to write your own wrapper function in Python, etc. that loads the model into your inference engine of choice encodes input images, runs inference, and decodes the output images.