Cannot start Metric 3D ONNX ROS2 inference

[sp4rq-dev]

Hi, I am able to run detect_seg_depth.launch.xml and it works very well but when I try to launch metric_3d.launch.xml to test Metric3D model I get the following error.

ros2 launch ros2_vision_inference metric_3d.launch.xml
[INFO] [launch]: All log files can be found below /home/sp4rq/.ros/log/2024-07-03-02-26-22-066362-legion-40304
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [vision_inference_node-1]: process started with pid [40305]
[vision_inference_node-1] [INFO] [1719966384.272657571] [vision_inference]: Reading parameters...
[vision_inference_node-1] [INFO] [1719966384.274854106] [vision_inference]: Initializing model...
[vision_inference_node-1] 2024-07-03 02:26:24.873300253 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 9 Memcpy nodes are added to the graph torch_jit for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
[vision_inference_node-1] 2024-07-03 02:26:24.881290749 [W:onnxruntime:, session_state.cc:1166 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
[vision_inference_node-1] 2024-07-03 02:26:24.881310445 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
[vision_inference_node-1] [INFO] [1719966385.969359040] [vision_inference]: Model Done
[vision_inference_node-1] [INFO] [1719966385.969816855] [vision_inference]: Initializing static memory...
[vision_inference_node-1] [INFO] [1719966385.979125203] [vision_inference]: Initialization Done
[vision_inference_node-1] /home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py:202: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
[vision_inference_node-1]   'mask' : mask.astype(np.bool)[None] # 1, H, W
[vision_inference_node-1] Exception in thread Thread-3:
[vision_inference_node-1] Traceback (most recent call last):
[vision_inference_node-1]   File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
[vision_inference_node-1]     self.run()
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 150, in run
[vision_inference_node-1]     onnx_input, pad_info = self.prepare_input(self.image)
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 202, in prepare_input
[vision_inference_node-1]     'mask' : mask.astype(np.bool)[None] # 1, H, W
[vision_inference_node-1]   File "/home/sp4rq/.local/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
[vision_inference_node-1]     raise AttributeError("module {!r} has no attribute "
[vision_inference_node-1] AttributeError: module 'numpy' has no attribute 'bool'. Did you mean: 'bool_'?
[vision_inference_node-1] segmentation runtime: 0.8024070262908936
[vision_inference_node-1] mono3d runtime: 1.7138075828552246
[vision_inference_node-1] Traceback (most recent call last):
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/install/ros2_vision_inference/lib/ros2_vision_inference/vision_inference_node", line 33, in <module>
[vision_inference_node-1]     sys.exit(load_entry_point('ros2-vision-inference', 'console_scripts', 'vision_inference_node')())
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 365, in main
[vision_inference_node-1]     VisionInferenceNode()
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 219, in __init__
[vision_inference_node-1]     self.ros_interface.spin()
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/utils/ros_util.py", line 46, in spin
[vision_inference_node-1]     rclpy.spin(self)
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/__init__.py", line 222, in spin
[vision_inference_node-1]     executor.spin_once()
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 739, in spin_once
[vision_inference_node-1]     self._spin_once_impl(timeout_sec)
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 736, in _spin_once_impl
[vision_inference_node-1]     raise handler.exception()
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/task.py", line 239, in __call__
[vision_inference_node-1]     self._handler.send(None)
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 437, in handler
[vision_inference_node-1]     await call_coroutine(entity, arg)
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 362, in _execute_subscription
[vision_inference_node-1]     await await_or_execute(sub.callback, msg)
[vision_inference_node-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 107, in await_or_execute
[vision_inference_node-1]     return callback(*args)
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 361, in camera_callback
[vision_inference_node-1]     self.process_image(image)
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 306, in process_image
[vision_inference_node-1]     depth = self.monodepth_thread.join() if self.monodepth_flag else None
[vision_inference_node-1]   File "/home/sp4rq/colcon_ws/build/ros2_vision_inference/ros2_vision_inference/ros2_vision_inference.py", line 93, in join
[vision_inference_node-1]     return self._output
[vision_inference_node-1] AttributeError: 'Metric3DThread' object has no attribute '_output'
[ERROR] [vision_inference_node-1]: process has died [pid 40305, exit code 1, cmd '/home/sp4rq/colcon_ws/install/ros2_vision_inference/lib/ros2_vision_inference/vision_inference_node --ros-args -r __node:=vision_inference --params-file /tmp/launch_params_c0n6y8h8 --params-file /tmp/launch_params_gohdyigu --params-file /tmp/launch_params_aa1w2cko --params-file /tmp/launch_params_oimac40g --params-file /tmp/launch_params_oxepg5r9 --params-file /tmp/launch_params_w7gn9uul --params-file /tmp/launch_params_rjvxb_me --params-file /tmp/launch_params_iklzg7n6 --params-file /tmp/launch_params__4e6_fpg --params-file /tmp/launch_params_yywom5ym --params-file /tmp/launch_params_fdptwjkm -r /image_raw:=/kitti/camera_color_left/image_raw -r /compressed_image:=/image/compressed -r /camera_info:=/kitti/camera_color_left/camera_info -r /bboxes:=/bboxes -r /point_cloud:=/point_cloud'].

Below is my lauch file metric_3d.launch.xml:

<?xml version="1.0"?>
<launch>
    <arg name="mono3d_flag" default="true"/>
    <arg name="mono3d_ckpt_file" default="/home/sp4rq/models/det3d/dla34_deform_576_768.onnx"/>
    <arg name="mono3d_gpu_index" default="0"/>

    <arg name="seg_flag" default="true"/>
    <arg name="seg_ckpt_file" default="/home/sp4rq/models/segmentation/bisenetv1.onnx"/>
    <arg name="seg_gpu_index" default="0"/>

    <arg name="monodepth_flag" default="true"/>
    <arg name="monodepth_ckpt_file" default="/home/sp4rq/models/depth/metric_3d.onnx"/>
    <arg name="monodepth_gpu_index" default="0"/>
    <arg name="monodepth_use_metric_3d" default="true"/>

    <arg name="camera_topic" default="/kitti/camera_color_left/image_raw"/>
    <arg name="compressed_image_topic" default="/image/compressed"/>
    <arg name="camera_param" default="/kitti/camera_color_left/camera_info"/>
    <arg name="output_bbox_topic" default="/bboxes"/>
    <arg name="output_seg_pointcloud_topic" default="/point_cloud"/>

    <node pkg="ros2_vision_inference" exec="vision_inference_node" name="vision_inference" output="screen">
        <param name="MONO3D_FLAG" value="$(var mono3d_flag)"/>
        <param name="MONO3D_CKPT_FILE" value="$(var mono3d_ckpt_file)"/>
        <param name="MONO3D_GPU_INDEX" value="$(var mono3d_gpu_index)"/>

        <param name="SEG_FLAG" value="$(var seg_flag)"/>
        <param name="SEG_CKPT_FILE" value="$(var seg_ckpt_file)"/>
        <param name="SEG_GPU_INDEX" value="$(var seg_gpu_index)"/>

        <param name="MONODEPTH_FLAG" value="$(var monodepth_flag)"/>
        <param name="MONODEPTH_CKPT_FILE" value="$(var monodepth_ckpt_file)"/>
        <param name="MONODEPTH_GPU_INDEX" value="$(var monodepth_gpu_index)"/>
        <param name="MONODEPTH_USE_METRIC_3D" value="$(var monodepth_use_metric_3d)"/>

        <param name="opacity" value="0.4"/>
        <remap from="/image_raw" to="$(var camera_topic)"/>
        <remap from="/compressed_image" to="$(var compressed_image_topic)"/>
        <remap from="/camera_info" to="$(var camera_param)"/>
        <remap from="/bboxes" to="$(var output_bbox_topic)"/>
        <remap from="/point_cloud" to="$(var output_seg_pointcloud_topic)"/>
    </node>

</launch>

[Owen-Liuyuxuan]

This is related to numpy version. You could change all np.bool to np.bool_ to solve the problem.

[sp4rq-dev]

This is related to numpy version. You could change all np.bool to np.bool_ to solve the problem.

Thanks a lot, now everything works. I would like to ask what license this repository is under. Can I make a fork and add some changes for learning and research purposes?

[Owen-Liuyuxuan]

@sp4rq-dev This is the work I mainly did when I was a student. So feel free to play with the repository.

[sp4rq-dev]

@Owen-Liuyuxuan Great, thanks!