search

IntelRealSense / realsense-ros

ROS Wrapper for Intel(R) RealSense(TM) Cameras
http://wiki.ros.org/RealSense
Apache License 2.0
2.43k stars 1.71k forks source link

Failing to publish compressed image and depth topic data on Nvidia AGX Orin #3105

Open nwissner-bdai opened 1 month ago

nwissner-bdai commented 1 month ago
Required Info
Camera Model D455
Firmware Version R36 (release), REVISION: 2.0
Operating System & Version Linux (Ubuntu 22.04.4 LTS)
Kernel Version (Linux Only) 6.5.0-28-generic
Platform Nvidia Jetson AGX Orin
Librealsense SDK Version v2.54.2
Language C++/Python/ROS2
Segment Robot
ROS Distro Humble
RealSense ROS Wrapper Version 4.54.1

Issue Description

When running the realsense2_camera, the camera/camera/color/image_raw and camera/camera/depth/image_rect_raw topics are echoing data just fine, but when trying to echo any data on the compressed topics, I am getting errors such as:

[realsense2_camera_node-1] [ERROR] [1715629235.416347303] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 170425562634720 bytes in function 'OutOfMemoryError'

When echoing image_raw/compressed data, and:

[realsense2_camera_node-1] [ERROR] [1715629297.743316357] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

When echoing image_rect_raw/compressed data.

MartyG-RealSense commented 1 month ago

Hi @nwissner-bdai May I please confirm that you are using the echo command below to access image_raw/compressed.

ros2 topic echo /camera/camera/color/image_raw/compressed

To access the compressed topics, the image-transport plugin must have been installed for Humble with the command below.

sudo apt install ros-humble-image-transport

nwissner-bdai commented 1 month ago

Hello @MartyG-RealSense , yes I am using ros2 topic echo etc. as a command to listen for compressed image or depth data. I also verified that ros-humble-image-transport was installed when I got these errors

dpkg -l | grep ros-humble-image-transport
ii  ros-humble-image-transport                         3.1.9-1jammy.20240327.160923                arm64        image_transport should always be used to subscribe to and publish images.
ii  ros-humble-image-transport-plugins                 2.5.1-1jammy.20240415.141447                arm64        A set of plugins for publishing and subscribing to sensor_msgs/Image topics in representations other than raw pixel data.
annb3 commented 1 month ago

@nwissner-bdai I have the same issue and the same version packages launching dpkg -l | grep ros-humble-image-transport.

MartyG-RealSense commented 1 month ago

The errors [16UC1] is not a color format. but [mono8] is and error: (-4:Insufficient memory) have not been previously reported in the past by other RealSense ROS wrapper users.

@nwissner-bdai and @annb3 Which RealSense firmware version are your cameras using, please? When using librealsense SDK 2.54.1 the camera firmware should be 5.15.0.2 and when using SDK 2.54.2 it should be 5.15.1.0. Using firmware versions other than the recommended one for a particular librealsense version can cause errors.

annb3 commented 1 month ago

@MartyG-RealSense I have SDK 2.55.1 and firmware version 5.16.01 but I have always the problem: [CompressedPublisher]: OpenCV(4.9.0-dev) /home/kitt/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 279965582913600 bytes in function 'OutOfMemoryError' [realsense2_camera_node-33] (publish() at ./src/compressed_publisher.cpp:240)

MartyG-RealSense commented 1 month ago

@annb3 The current ROS wrapper 4.54.1 is not designed for use with SDK 2.55.1 and firmware 5.16.0.1. There is not currently a wrapper for 2.55.1. I note though that @nwissner-bdai has been using 2.54.2 and still also experienced 'OutOfMemoryError'.

Do the compressed topic echoing errors still occur if you use a low resolution and FPS of 640x480 and 15 FPS?

ros2 launch realsense2_camera rs_launch.py depth_module.depth_profile:=640x480x15

nwissner-bdai commented 1 month ago

@MartyG-RealSense So the fw for the realsense camera I was using was not updated to 5.15.1.0, so I updated it and tested again to echo the compressed image topic data. Unfortunately still receiving the same errors. I also added the launch arguments you described and same errors. Do you recommend any other troubleshooting steps before I dig further into the problem?

annb3 commented 1 month ago

@nwissner-bdai I have also 2.55.1 and 5.16.0.1 and ROS wrapper works perfectly- After some experiments I have been able to solve the problem in this way:

This solution works in my Jetson Orin. Let me know if you will solve in this way or will you find some other solutions..

MartyG-RealSense commented 1 month ago

Thanks so much @annb3 for sharing the difference that enabling the pointcloud filter made for you in your particular situation!

Yes, please do let us know @nwissner-bdai if the above method works for you.

Rebuilding the wrapper may not be a necessary step, as the launch file is independent from the wrapper files and is not affected by a rebuild.

Instead of editing and saving the launch file, the launch command below may produce the same outcome.

ros2 launch realsense2_camera rs_launch.py depth_module.depth_profile:=640x480x15 pointcloud.enable:=true

annb3 commented 1 month ago

@MartyG-RealSense I use the file inside an enourmous repository of concatenate launchers and so on. that's why I can't launch the camera in this way and I have to rebuild (need to modify general settings of the overall project and it is super fast). This is the only possible solution for my purpose.

MartyG-RealSense commented 1 month ago

Thanks very much @annb3 for the clarification about your situation.

nwissner-bdai commented 1 month ago

@MartyG-RealSense Okay after doing some more testing, I have found that adding that additional launch argument you provided still produces the same errors I have been getting. Like @annb3, I too am working with an enormous repository made up of various research projects and am tasked with debugging this specific issue of publishing high resolution compressed image and depth data. My original testing involved launching a node from the repo which also spawned rs_launch.py, but now I have ran the rs_launch.py as a standalone node with and without your launch arguments and am still getting the same console errors when trying to ros2 topic echo /compressionTopicEtc. Specifically:

ros2 topic echo /camera/camera/depth/image_rect_raw/compressed
[realsense2_camera_node-1] [ERROR] [1716407232.444214183] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

and:

ros2 topic echo /camera/camera/color/image_raw/compressed
[realsense2_camera_node-1] [ERROR] [1716407167.873317761] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 118739372698080 bytes in function 'OutOfMemoryError'

I am now going to test this standalone node on non agx-orin hardware (thinkpad) and will update with those results when I can. I'll be sure to double check versioning for the sdk, wrapper and firmware on the camera

MartyG-RealSense commented 1 month ago

Thanks very much @nwissner-bdai for the detailed update. I look forward to your next report after further testing. Good luck!

nwissner-bdai commented 4 weeks ago

Hello @MartyG-RealSense , I have tested Librealsense SDK versions paired with their appropriate realsense-ros wrapper versions on both the AGX Orin and Lenovo Thinkpad platforms extensively, documenting the topic statistics with ros2 topic hz and ros2 topic bw. I've found that there hasnt been a reliable way to publish compressed image and depth data on the AGX Orin platform for the last 3 major releases, which means this may require some further investigation into the root cause of the issue. I have also collected all the errors that are present when trying to echo the topic data. I will attach all my findings below:

Lenovo Thinkpad

SDK Software Version and Firmware Version

Realsense-ros 4.54.1
Librealsense 2 SDK 2.54.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100 msgs)
/camera/color/image_raw 11.455 29.38
/camera/camera/color/image_raw/compressed 14.987 1.25
/camera/camera/color/image_raw/theora 15.525 22.00
/camera/camera/depth/image_rect_raw 7.041 4.14
/camera/camera/depth/image_rect_raw/compressedDepth 3.524 275.53

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1716919035.770059648] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1716919161.666898386] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1716919219.511761717] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

SDK Software Version and Firmware Version

Realsense-ros 4.51.1
Librealsense 2 SDK 2.51.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw 20.258 52.96
/camera/color/image_raw/compressed 29.992 7.82
/camera/color/image_raw/theora 23.770 7.77
/camera/depth/image_rect_raw 8.766 10.05
/camera/depth/image_rect_raw/compressedDepth 2.204 172.10

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressedDepth

 [realsense2_camera_node-1] [ERROR] [1717440347.780579745] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/depth/image_rect_raw/compressed (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717440501.576116078] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed 14.996 1.27

SDK Software Version and Firmware Version

Realsense-ros 4.55.1
Librealsense 2 SDK 2.55.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw 18.351 56.23
/camera/color/image_raw/compressed 29.976 8.66
/camera/color/image_raw/theora 22.720 15.08
/camera/depth/image_rect_raw 20.233 17.07
/camera/depth/image_rect_raw/compressedDepth 7.295 383.56 (KB/s)

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717446915.676193022] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717447086.328406218] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717447310.293408762] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed 29.986 2.52 (KB/s)
/camera/depth/image_rect_raw/compressedDepth 1.049 212.87 (KB/s)

AGX Orin

SDK Software Version and Firmware Version

Realsense-ros 4.54.1
Librealsense 2 SDK 2.54.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw 29.850 83.70
/camera/depth/image_rect_raw 22.278 19.96
/camera/depth/image_rect_raw/compressedDepth 1.172 214.15 (KB/s)

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717098382.214202425] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 222112060848000 bytes in function 'OutOfMemoryError'

/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717098440.072736737] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/color/image_raw/theora (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717098497.831257488] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 222111994788720 bytes in function 'OutOfMemoryError'

/camera/depth/image_rect_raw/compressed (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717098869.836639329] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/depth/image_rect_raw/compressedDepth (data printing w/ error)

[realsense2_camera_node-1]  30/05 15:56:45,445 ERROR [281472207808736] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 130

/camera/depth/image_rect_raw/theora (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717099131.918398786] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed 29.977 2.54 (KB/s)
/camera/depth/image_rect_raw/compressedDepth 1.049 212.87 (KB/s)

SDK Software Version and Firmware Version

Realsense-ros 4.51.1
Librealsense 2 SDK 2.51.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw 29.96 66.64
/camera/depth/image_rect_raw 27.993 24.56
/camera/depth/image_rect_raw/compressedDepth 1.158 209.09

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717201094.335925484] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 180141772297680 bytes in function 'OutOfMemoryError'

/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717201166.633408758] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/color/image_raw/theora

[realsense2_camera_node-1] [ERROR] [1717201199.114697728] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 180141706238400 bytes in function 'OutOfMemoryError'

/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717201457.870049961] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717201679.300377455] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

SDK Software Version and Firmware Version

Realsense-ros 4.55.1
Librealsense 2 SDK 2.55.1
D455 Firmware 5.16.0.1

Successful Image and Compression Topics

Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/camera/color/image_raw 29.948 73.29
/camera/camera/depth/image_rect_raw 29.959 29.242
/camera/camera/depth/image_rect_raw/compressedDepth 1.041 211.78

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressed (data printing w/ errors)

[realsense2_camera_node-1] [ERROR] [1717443390.451790943] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 293174478828720 bytes in function 'OutOfMemoryError'

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717443507.035844806] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/color/image_raw/theora

[realsense2_camera_node-1] [ERROR] [1717443535.281818186] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 293174412769440 bytes in function 'OutOfMemoryError'

/camera/camera/depth/image_rect_raw/compressed (data printing w/ errors)

[realsense2_camera_node-1] [ERROR] [1717443679.953734063] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717443926.629635525] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
Topic Name ros2 topic hz ros2 topic bw (MB/s @ 100msgs)
/camera/camera/color/image_raw/compressed 29.923 2.4 (KB/s)
/camera/camera/depth/image_rect_raw/compressed 29.985 2.53 (KB/s)

Apologies if this information doesn't prove to be as useful as I had hoped, but it would be nice if the the community could take a special interest in this issue as it seems to be repeatable and prevalent to other users such as @annb3. If there is any other data you would recommend I collect on this issue I am happy to further investigate!

MartyG-RealSense commented 4 weeks ago

Thank you very much for the detailed test results. It appears that for both the Lenovo and AGX, /camera/depth/image_rect_raw/compressedDepth is the topic that has the most problems. This may be because the topic is using a very high amount of bandwidth MB compared to the other compressed depth topic /camera/camera/color/image_raw/compressed which has very good performance and small bandwidth usage in comparison.

There was a past case at https://github.com/IntelRealSense/realsense-ros/issues/1672 where a RealSense user also experienced low performance from /camera/depth/image_rect_raw/compressedDepth and they were able to make it faster by changing the value of a setting called png_level.

nwissner-bdai commented 4 weeks ago

@MartyG-RealSense Thank you for taking such consideration to look into this! I would say /camera/color/image_raw/compressed and /camera/color/image_raw/compressedDepth are the most inconsistent and problematic in the case of the AGX Orin which is the focus of my problem, was only using the Thinkpad as a control. camera/depth/image_rect_raw/compressedDepth at least successfully publishes data in most if not all cases. I did test trying lower values for the png_level parameter but saw marginal difference in bandwidth for the camera/depth/image_rect_raw/compressedDepth topic. Is there any plan on fixing this issue should it occur with other users? I've tried editing the other parameters suggested and am still in no luck of publishing the compressed topics I mentioned earlier

nwissner-bdai commented 3 weeks ago

@MartyG-RealSense Oh I am working with ROS2 and not ROS1 for context. I do have the ros-humble-image-transport and ros-humble-image-transport-plugins correctly installed

MartyG-RealSense commented 3 weeks ago

Thank you @nwissner-bdai - I will highlight this issue with high bandwidth usage to my Intel colleagues on the RealSense ROS team.

Edit: the ROS team will attempt to look at this issue at a future date when there is the opportunity to work on it.

annb3 commented 3 weeks ago

@nwissner-bdai I can confirm that actually somehow I again receive the error: [CompressedPublisher]: OpenCV(4.9.0-dev) /home/kitt/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 279965582913600 bytes in function 'OutOfMemoryError' [realsense2_camera_node-33] (publish() at ./src/compressed_publisher.cpp:240) even if I have already modified the code, so there is something more to be considered. @MartyG-RealSense I suggest to go more deeper testing Release on Orin platform.

nwissner-bdai commented 5 days ago

Hello @MartyG-RealSense , I was just wondering if the ROS team had made any progress on this issue that I could report back to my team as this is a blocking issue for them

MartyG-RealSense commented 4 days ago

Hi @nwissner-bdai I will follow up with the RealSense ROS team about this issue.

MartyG-RealSense commented 1 day ago

Hi @nwissner-bdai My Intel RealSense colleagues have officially created an internal ticket so that they can look at this issue with compression bandwidth at a future date when there is the opportunity to work on it.

nwissner-bdai commented 1 day ago

Sounds good, thank you for getting back to me @MartyG-RealSense