search

NVIDIA-ISAAC-ROS / isaac_ros_visual_slam

Visual SLAM/odometry package based on NVIDIA-accelerated cuVSLAM
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
867 stars 139 forks source link

will it work with zed2? Jetson, Jetpack 4.6.2. zed sdk 3.6.1 #8

Closed AndreV84 closed 2 years ago

AndreV84 commented 2 years ago

Will it work with the zed2? The Elbrus? https://forums.developer.nvidia.com/t/will-it-work-with-the-zed2-the-elbrus/196452

hemalshahNV commented 2 years ago

Yes, Elbrus SVIO has been used successfully with ZED2.

From: Andrey Volodin @.> Sent: Sunday, November 28, 2021 8:10:56 AM To: NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry @.> Cc: Subscribed @.***> Subject: [NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry] will it work with zed2? (Issue #8)

Will it work with the zed2? The Elbrus?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry/issues/8, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASS44RWHEYK2RHRNJVIWNWTUOJIBBANCNFSM5I5OOPFA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

AndreV84 commented 2 years ago

Yes, Elbrus SVIO has been used successfully with ZED2.

Thank you for your reply! so deployingf to jetson 4.6 with jetpack4.5 argument to try on jetson with zed2? It would require to buils the sample app somehow? how to build it with analogy to 

bazel build //apps/samples/zed_camera

there is no elbrus sample app at the Host PC

./../engine/engine/build/deploy.sh --remote_user user -p //apps/samples/elbrus_visual_slam_zed:elbrus_visual_slam_zed-pkg -d jetpack45 -h 192.168.x.x

jetpack 4.6 argument doesn't seem supported for jetson deployment? also can not find as for now deployment line for the elbrus zed svo Ffrom the instruction:

The “elbrus_visual_slam_zed” sample application uses the ZED camera, which performs software undistortion inside the StereoLabs SDK.

but there is no such app in the sample app folder It only has

april_tags    follow_me           imu_calibration      manipulation          pwm               stereo_rectification  vicon_mocap
argus_camera  hgmm_matching       imu_sim              navigation_rosbridge  python_api        subprocess            zed_camera
BUILD         image_undistortion  __init__.py          pick_and_place        realsense_camera  teleport_sim
camera        imu                 interactive_markers  proto_to_json         stereo_dummy      v4l2_camera
sdk/packages_jetpack45/visual_slam$ ls
BUILD  libvisual_slam_module.so

/sdk/packages_x86_64_cuda_11_1/visual_slam$ ls
BUILD  libvisual_slam_module.so
AndreV84 commented 2 years ago

where to get these files on jetson? packages/visual_slam/apps/elbrus_visual_slam_zed_python.py trying to build on the Host PC with

bazel build packages/visual_slam/apps/elbrus_visual_slam_zed

@hemalshahNV but how to deploy it to jetson once built?

hemalshahNV commented 2 years ago

Are you using Elbrus in Isaac SDK? If so, you’ll want to post in our dev forums so other Isaac SDK users can help.

From: Andrey Volodin @.> Sent: Sunday, November 28, 2021 10:27:32 AM To: NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry @.> Cc: Hemal Shah @.>; Comment @.> Subject: Re: [NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry] will it work with zed2? (Issue #8)

where to get these files? IsaaC sdk downloaded doesn't seem having it packages/visual_slam/apps/elbrus_visual_slam_zed_python.py

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry/issues/8#issuecomment-981129988, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASS44RRE4UFIC3H5WXVPKI3UOJYBJANCNFSM5I5OOPFA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

AndreV84 commented 2 years ago

@hemalshahNV Thank you for your response: "Are you using Elbrus in Isaac SDK?" No, I used the ROS node with realsense without the IsaaC. However as you confirmed that zed2 works wit hElbrus, coudl you extend if it is supported in a standalone mode [ like with realsense] or with Isaac + Elbrus mode? Both? neither of the two? Moreover, the linked reference thread is at the first message in the github issues post https://forums.developer.nvidia.com/t/will-it-work-with-the-zed2-the-elbrus/196452

AndreV84 commented 2 years ago

anyway posted at both places, could you extend on steps to run zed2 with elbrus anyhow please e.g. in the setup you confirmed previously, please?

hemalshahNV commented 2 years ago

I'm afraid I'm not following what you're trying to do exactly, but let me see if I can help on the ROS-side of the conversation here and address the Isaac SDK part of it in the Dev Forum post you made (my bad, did not notice the link in your first post). You would first have to go prepare the Zed2 ROS2 packages here. Then, there is an Isaac ROS launch file that should bring up the VO nodes with topics remapped for Zed. All you would have to do is launch the ZED camera ROS2 node then launch this Isaac ROS VO launch file and you should be up and running. We had configured the Realsense version of this launch file to also bring up the Realsense camera node as well but didn't have a chance to do something similar for ZED though, so we had to leave launching the ZED camera node up to the user.

AndreV84 commented 2 years ago

@hemalshahNV Thank you for your reply. I will be trying to setup the zed2ros2 package to run it on jetson in foxy2 container docker implementation provided by you. But I will have to separately copy the file https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry/blob/main/isaac_ros_visual_odometry/launch/isaac_ros_visual_odometry_zed2.launch.py manually to Jetson, right? or it won't work with just copying the single file but would need to deploy also the environment with Isaac[bazel] from Host PC to Jetson?

AndreV84 commented 2 years ago

@hemalshahNV after installing zed2 srapper copying pyhton script getting this on ros2 foxy jetson

after running zed2 node.:

ros2 launch zed_wrapper zed2.launch.py

how to run the python file

ros2 launch isaac_ros_visual_odometry isaac_ros_visual_odometry_zed2.launch.py

like that?

 ros2 topic echo /visual_odometry/tf_stamped

However the output is empty of the latter. Any ideas?

AndreV84 commented 2 years ago 
 ros2 launch isaac_ros_visual_odometry isaac_ros_visual_odometry_zed2.launch.py 
[INFO] [launch]: All log files can be found below /home/user/.ros/log/-------
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [198027]
[component_container-1] [INFO] [1638199221.682004070] [visual_odometry_launch_container]: Load Library: /home/user/your_ws/src/install/isaac_ros_visual_odometry/lib/libvisual_odometry_node.so
[component_container-1] [INFO] [1638199221.959088912] [visual_odometry_launch_container]: Found class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_odometry::VisualOdometryNode>
[component_container-1] [INFO] [1638199221.959244050] [visual_odometry_launch_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_odometry::VisualOdometryNode>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/visual_odometry_node' in container '/visual_odometry_launch_container'
hemalshahNV commented 2 years ago

I will be trying to setup the zed2ros2 package to run it on jetson in foxy2 container docker implementation provided by you. But I will have to separately copy the file https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry/blob/main/isaac_ros_visual_odometry/launch/isaac_ros_visual_odometry_zed2.launch.py manually to Jetson, right? or it won't work with just copying the single file but would need to deploy also the environment with Isaac[bazel] from Host PC to Jetson?

You will need to checkout the Isaac ROS VO repository to your Jetson and build the packages within using the ROS2 standard colcon. There is no Bazel involved. Your steps in the subsequent post look right. You may want to check that the topics are all mapped correctly with ros2 topic list and ros2 topic info to make sure everything is wired appropriately.

AndreV84 commented 2 years ago

@hemalshahNV Thank you for your reply! It seems I built all components on jetson. right?

colcon build --symlink-install && source install/setup.bash
Starting >>> isaac_ros_common
Starting >>> isaac_ros_test
Starting >>> zed_interfaces                                                         
Starting >>> isaac_ros_nvengine_interfaces
Starting >>> isaac_ros_visual_odometry_interfaces
Finished <<< isaac_ros_common [2.72s]                                                                                             
Finished <<< isaac_ros_visual_odometry_interfaces [4.44s]                                        
Finished <<< isaac_ros_test [4.89s]                                                                                                         
Starting >>> isaac_ros_image_proc
Starting >>> isaac_ros_stereo_image_proc                                                                                
Finished <<< isaac_ros_nvengine_interfaces [4.98s]                                                                   
Starting >>> isaac_ros_nvengine
Finished <<< isaac_ros_image_proc [1.90s]                                                                      
Starting >>> isaac_ros_visual_odometry
Finished <<< isaac_ros_nvengine [1.82s]                                                                          
Finished <<< zed_interfaces [7.26s]                                                                              
Starting >>> zed_components
Finished <<< isaac_ros_visual_odometry [1.24s]                                                                                 
Finished <<< zed_components [1.09s]                                                                          
Starting >>> zed_wrapper                                         
Finished <<< zed_wrapper [0.85s]                                                                             
Starting >>> zed_ros2
Finished <<< zed_ros2 [0.65s]                                                                            
Finished <<< isaac_ros_stereo_image_proc [14.6s]                         
Starting >>> isaac_ros_image_pipeline
Finished <<< isaac_ros_image_pipeline [0.63s]                     

Summary: 13 packages finished [21.1s]

The topic list shows the below.

~/your_ws$ ros2 topic list 
/diagnostics
/parameter_events
/rosout
/tf
/tf_static
/visual_odometry/tf_stamped
/zed2/joint_states
/zed2/robot_description
/zed2/zed_node/atm_press
/zed2/zed_node/confidence/confidence_map
/zed2/zed_node/depth/camera_info
/zed2/zed_node/depth/depth_registered
/zed2/zed_node/disparity/disparity_image
/zed2/zed_node/imu/data
/zed2/zed_node/imu/data_raw
/zed2/zed_node/imu/mag
/zed2/zed_node/left/camera_info
/zed2/zed_node/left/image_rect_color
/zed2/zed_node/left/image_rect_gray
/zed2/zed_node/left_cam_imu_transform
/zed2/zed_node/left_raw/camera_info
/zed2/zed_node/left_raw/image_raw_color
/zed2/zed_node/left_raw/image_raw_gray
/zed2/zed_node/odom
/zed2/zed_node/path_map
/zed2/zed_node/path_odom
/zed2/zed_node/point_cloud/cloud_registered
/zed2/zed_node/pose
/zed2/zed_node/pose_with_covariance
/zed2/zed_node/rgb/camera_info
/zed2/zed_node/rgb/image_rect_color
/zed2/zed_node/rgb/image_rect_gray
/zed2/zed_node/rgb_raw/camera_info
/zed2/zed_node/rgb_raw/image_raw_color
/zed2/zed_node/rgb_raw/image_raw_gray
/zed2/zed_node/right/camera_info
/zed2/zed_node/right/image_rect_color
/zed2/zed_node/right/image_rect_gray
/zed2/zed_node/right_raw/camera_info
/zed2/zed_node/right_raw/image_raw_color
/zed2/zed_node/right_raw/image_raw_gray
/zed2/zed_node/stereo/image_rect_color
/zed2/zed_node/stereo_raw/image_raw_color
/zed2/zed_node/temperature/imu
/zed2/zed_node/temperature/left
/zed2/zed_node/temperature/right

the topic info shows 

ros2 topic info /visual_odometry/tf_stamped
Type: isaac_ros_visual_odometry_interfaces/msg/VisualOdometryTransformStamped
Publisher count: 1
Subscription count: 1

zed topics have the output, but the visual odometry topic is silent

AndreV84 commented 2 years ago 

~/your_ws/src$ ls
isaac_ros_common  isaac_ros_image_pipeline  isaac_ros_visual_odometry  zed-ros2-wrapper
****
AndreV84 commented 2 years ago

does zed2 need certain firmware? sdk version? I just installed the latest available sdk from stereolabs website. zed2 node seems working with it as ther is e..g pose topic poblished does the phrase below reffer to docker container? 

 Loaded node '/visual_odometry_node' in container '/visual_odometry_launch_container

The setup is systemwide, no container is used

AndreV84 commented 2 years ago

after removing src folder then just cloning zed wrapper/ odometry then building just two packages


cat log.txt 
In file included from /home/user/your_ws/install/zed_components/include/zed_components/sl_tools.h:50,
                 from /home/user/your_ws/install/zed_components/include/zed_components/zed_camera_component.hpp:29,
                 from /home/user/your_ws/zed-ros2-wrapper/zed_wrapper/src/zed_wrapper.cpp:26:
/usr/local/zed/include/sl/Camera.hpp:784:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:937:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1114:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1278:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1282:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1286:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1526:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1530:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp:1534:13: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
             };
             ^
/usr/local/zed/include/sl/Camera.hpp: In function ‘cudaError sl::__cudaSafeCall(cudaError, const char*, const char*, int)’:
/usr/local/zed/include/sl/Camera.hpp:2048:89: warning: unused parameter ‘file’ [-Wunused-parameter]
 daError __cudaSafeCall(cudaError err, const char *func, const char *file, const int line) {
                                                         ~~~~~~~~~~~~^~~~

/usr/local/zed/include/sl/Camera.hpp:2048:105: warning: unused parameter ‘line’ [-Wunused-parameter]
 daSafeCall(cudaError err, const char *func, const char *file, const int line) {
                                                               ~~~~~~~~~~^~~~

In file included from /home/user/your_ws/install/zed_components/include/zed_components/sl_tools.h:50,
                 from /home/user/your_ws/install/zed_components/include/zed_components/zed_camera_component.hpp:29,
                 from /home/user/your_ws/zed-ros2-wrapper/zed_wrapper/src/zed_wrapper.cpp:26:
/usr/local/zed/include/sl/Camera.hpp: At global scope:
/usr/local/zed/include/sl/Camera.hpp:5335:12: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
     inline const void /*@cond SHOWHIDDEN*/SL_SDK_EXPORT/*@endcond*/ getZEDSDKBuildVersion(int &major, int& minor, int& patch) {
            ^~~~~
In file included from /home/user/your_ws/install/zed_components/include/zed_components/sl_tools.h:50,
                 from /home/user/your_ws/install/zed_components/include/zed_components/zed_camera_component.hpp:29,
                 from /home/user/your_ws/zed-ros2-wrapper/zed_wrapper/src/zed_wrapper.cpp:26:
/usr/local/zed/include/sl/Camera.hpp:5724:2: warning: extra ‘;’ [-Wpedantic]
 };
  ^
hemalshahNV commented 2 years ago

Are you seeing any traffic on topic /zed2/zed_node/left/image_rect_gray? (ros2 topic hz /zed2/zed_node/left/image_rect_gray) The isaac_ros_visual_odometry_zed2.launch.py launch file should work remap to the topics coming from the Zed camera in your ros2 topic list, so everything should be wired together. We need to confirm that Zed camera is actually outputting anything.

AndreV84 commented 2 years ago

@hemalshahNV the p;ose works

ros2 topic hz /zed2/zed_node/pose
average rate: 15.156

the rect grey seems empty

 ros2 topic hz /zed2/zed_node/left/image_rect_gray
AndreV84 commented 2 years ago

@hemalshahNV Thanks seems done by installing the package below https://github.com/ros-perception/image_common/archive/refs/tags/3.0.0.tar.gz


ros2 topic hz /visual_odometry/tf_stamped

average rate: 15.116
    min: 0.059s max: 0.073s std dev: 0.00334s window: 17
average rate: 15.146
    min: 0.059s max: 0.073s std dev: 0.00301s window: 33
average rate: 15.141
    min: 0.059s max: 0.073s std dev: 0.00284s window: 49
AndreV84 commented 2 years ago

so for zed2 there is no launch script for vizualization anyhow with issac sim? else? the most simple way to vizualise will be which from the text outputs for zed2? Thanks

AndreV84 commented 2 years ago

@hemalshahNV Could you confirm there is still no way to run elbrus on Jetson with basel? only with ros2? Thanks AV

AndreV84 commented 2 years ago

odometry package won't build anymore in container I need to test ros2 elbrus with zed somehow


/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:598:25: error: request for member ‘num_parameters’ in ‘elbrus_camera’, which is of non-class type ‘int’
           elbrus_camera.num_parameters = 4;
                         ^~~~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:605:25: error: request for member ‘distortion_model’ in ‘elbrus_camera’, which is of non-class type ‘int’
           elbrus_camera.distortion_model = kPinhole;
                         ^~~~~~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:606:25: error: request for member ‘num_parameters’ in ‘elbrus_camera’, which is of non-class type ‘int’
           elbrus_camera.num_parameters = 4;
                         ^~~~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp: At global scope:
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:618:1: error: ‘ELBRUS_Image’ does not name a type
 ELBRUS_Image VisualOdometryNode::VisualOdometryImpl::ToElbrusImage(
 ^~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:633:1: error: ‘ELBRUS_ImuMeasurement’ does not name a type
 ELBRUS_ImuMeasurement VisualOdometryNode::VisualOdometryImpl::ToElbrusImuMeasurement(
 ^~~~~~~~~~~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp: In destructor ‘isaac_ros::visual_odometry::VisualOdometryNode::VisualOdometryImpl::~VisualOdometryImpl()’:
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:742:7: error: ‘elbrus_handle’ was not declared in this scope
   if (elbrus_handle != nullptr) {ELBRUS_DestroyTracker(elbrus_handle);}
       ^~~~~~~~~~~~~
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:742:7: note: suggested alternative: ‘elbrus_cameras’
   if (elbrus_handle != nullptr) {ELBRUS_DestroyTracker(elbrus_handle);}
       ^~~~~~~~~~~~~
       elbrus_cameras
/workspaces/isaac_ros-dev/src/isaac_ros_visual_slam-0.9.1-ea2/isaac_ros_visual_odometry/src/visual_odometry_node.cpp:742:34: error: ‘ELBRUS_DestroyTracker’ was not declared in this scope
   if (elbrus_handle != nullptr) {ELBRUS_DestroyTracker(elbrus_handle);}
                                  ^~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/visual_odometry_node.dir/src/visual_odometry_node.cpp.o] Error 1
make[1]: *** [CMakeFiles/visual_odometry_node.dir/all] Error 2
make: *** [all] Error 2
AndreV84 commented 2 years ago

@hemalshahNV We managed to run zedm with prior version of the odometry However, it only seem to generate one topic as resulting output /visual_odometry/tf_stamped is that correct?

AndreV84 commented 2 years ago

@hemalshahNV does it have zed support by now? with the most recent release? Thanks

hemalshahNV commented 2 years ago

We have not developed any ZED support specifically but there is no reason that it should not work that we know of if you setup the ZED SDK and everything properly.

AndreV84 commented 2 years ago

@hemalshahNV Thank you for following up! However, while installing the SDK from stereolabs is not an issue at all, but a trivial task, the problem is to get zed stereolabs humble wrapper node installed in humble container supplied from you. Neither it seems to build even if to update the os to 22.04 to get apt ros humble support If you will have any luck with testing the zed coherence with slam or nvblox, let me know please

shtern94 commented 1 year ago

@hemalshahNV hello, Was isaac_ros_visual_odometry_zed2.launch.py deleted from the repository? I wonder what were the launch parameters for ZED and visual slam nodes. I tried making ZED camera work with Isaac Visual SLAM, but unfortunately I'm getting messages that the tracker got lost. I've remapped all the frames, tried disabling ZED's tf2, nothing helped..

hemalshahNV commented 1 year ago

@shtern94 , yes, we removed the ZED launch file because it was not being tested regularly at the time as changes to the Stereolabs SDK were coming through. In an upcoming release, we're working on adding more complete ZED support and currently debugging ZED and Isaac ROS Visual SLAM in fact. When we find anything, I'll report back.