DOPE network is not detecting training objects with real world images

[Maxime-Syc]

Hello everyone,

I am trying to develop a real world application utilizing the DOPE network. From my understanding, each of the 40 models can only detect one type of object each.

I am using a RealSense D435 as a camera and remapped the color/image_raw topic to /image so the DOPE network could access it and the image resolution was set to 640x480 at 30 fps.

The objects are laid on a white background with the camera 60.5 cm above, looking straight down.

I have 5 objects from the training data set (Heinz ketchup bottle, French mustard bottle, Campbells' tomato soup can and Cheez'in box) and none of their corresponding networks were detecting the objects. I got 1 or 2 "hits", but those seemed to be hallucinations has they appeared for a single frame and were nowhere near the object.

What could be the cause of those really poor performances? Inadequate lighting? Not having the exact object? (I tried to get as close as possible, but I live in Quebec, so the writing is usually in french and there are slight design variation)

Thank you for your help, I've been pulling my hair for almost 2 weeks now.

[swapnesh-wani-nvidia]

Hello, could you please provide the rosbag recording and configuration files to reproduce this issue.

[Maxime-Syc]

Hello,

Here is a link to a OneDrive folder containing the rosbags for all 7 tested DOPE models.

In the same folder, you will also find the parameter of the realsense2_camera_node in the file camera_configs.yaml. Here is the command line used to start the node: ros2 run realsense2_camera realsense2_camera_node --ros-args -r /color/image_raw:=/image -p rgb_camera.profile:=640x480x30

The -r /color/image_raw:=//image is required to forward the image feed of the camera to the topic DOPE is listening on, I haven't found where to configure the input topic.

Files are arranged in the following structure:

isaac_ros-dev
|-src
  |-isaac_ros_commun (submodule)
  |-isaac_ros_dnn_inference (submodule)
  |-isaac_ros_image_pipeline (submodule)
  |-isaac_ros_nitros (submodule)
  |-isaac_ros_pose_estimation (submodule)
  |-models
    |-cracker_60
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-Ketchup
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-meat_20
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-Mustard
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-mustard_60
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-soup_60
    | |-1
    | | |-model.onnx
    | |-config.pbtxt
    |-TomatoSauce
    | |-1
    | | |-model.onnx
    | |-config.pbtxt

The configuration files for the DOPE networks all follow the following form:

name: "{model_name}"
platform: "onnxruntime_onnx"
max_batch_size: 0
input [
  {
    name: "INPUT__0"
    data_type: TYPE_FP32
    dims: [ 1, 3, 480, 640 ]
  }
]
output [
  {
    name: "OUTPUT__0"
    data_type: TYPE_FP32
    dims: [ 1, 25, 60, 80 ]
  }
]
version_policy: {
  specific {
    versions: [ 1 ]
  }
}

Where {model_name} is replaced with the corresponding model name.

Here is a list of all seven commands used to launch DOPE with the different models:

• cracker_60: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=cracker_60 model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=cracker
• Ketchup: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=Ketchup model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=Ketchup
• meat_20: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=meat_20 model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=meat
• Mustard: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=Mustard model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=mustard
• mustard_60: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=mustard_60 model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=mustard
• soup_60: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=soup_60 model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=soup
• TomatoSauce: ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py model_name:=TomatoSauce model_repository_paths:=['/src/models'] input_binding_names:=['INPUT__0'] output_binding_names:=['OUTPUT__0'] object_name:=TomatoSauce

While recording these rosbags, I noticed the existence of the mustard_60 network in addition tot he Mustard model. mustard_60 performed significantly better than Mustard. I also realised that I was using the TomatoSauce to detect the tomato soup can instead of the visibly more appropriate soup_60 network.

When playing back the bags, you can visualize the images and the poses in RViz2 by adding the topics /image and /poses and setting the fixed frame to camera_color_frame.

I hope this is all the information you need and thank you for your assistance.

[jaiveersinghNV]

Hi @Maxime-Syc ,

We recently merged #23 which fixes some inconsistencies between the Python DOPE Training and Isaac ROS DOPE implementations.

Could you please take a look and see if the latest version of Isaac ROS DOPE corrects the issue you've observed?