nwissner-bdai commented 6 months ago

Required Info
Camera Model	D455
Firmware Version	R36 (release), REVISION: 2.0
Operating System & Version	Linux (Ubuntu 22.04.4 LTS)
Kernel Version (Linux Only)	6.5.0-28-generic
Platform	Nvidia Jetson AGX Orin
Librealsense SDK Version	v2.54.2
Language	C++/Python/ROS2
Segment	Robot
ROS Distro	Humble
RealSense ROS Wrapper Version	4.54.1

Issue Description

When running the realsense2_camera, the camera/camera/color/image_raw and camera/camera/depth/image_rect_raw topics are echoing data just fine, but when trying to echo any data on the compressed topics, I am getting errors such as:

[realsense2_camera_node-1] [ERROR] [1715629235.416347303] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 170425562634720 bytes in function 'OutOfMemoryError'

When echoing image_raw/compressed data, and:

[realsense2_camera_node-1] [ERROR] [1715629297.743316357] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

When echoing image_rect_raw/compressed data.

MartyG-RealSense commented 6 months ago

Hi @nwissner-bdai May I please confirm that you are using the echo command below to access image_raw/compressed.

ros2 topic echo /camera/camera/color/image_raw/compressed

To access the compressed topics, the image-transport plugin must have been installed for Humble with the command below.

sudo apt install ros-humble-image-transport

nwissner-bdai commented 6 months ago

Hello @MartyG-RealSense , yes I am using ros2 topic echo etc. as a command to listen for compressed image or depth data. I also verified that ros-humble-image-transport was installed when I got these errors

dpkg -l | grep ros-humble-image-transport
ii  ros-humble-image-transport                         3.1.9-1jammy.20240327.160923                arm64        image_transport should always be used to subscribe to and publish images.
ii  ros-humble-image-transport-plugins                 2.5.1-1jammy.20240415.141447                arm64        A set of plugins for publishing and subscribing to sensor_msgs/Image topics in representations other than raw pixel data.

annb3 commented 6 months ago

@nwissner-bdai I have the same issue and the same version packages launching dpkg -l | grep ros-humble-image-transport.

MartyG-RealSense commented 6 months ago

The errors [16UC1] is not a color format. but [mono8] is and error: (-4:Insufficient memory) have not been previously reported in the past by other RealSense ROS wrapper users.

@nwissner-bdai and @annb3 Which RealSense firmware version are your cameras using, please? When using librealsense SDK 2.54.1 the camera firmware should be 5.15.0.2 and when using SDK 2.54.2 it should be 5.15.1.0. Using firmware versions other than the recommended one for a particular librealsense version can cause errors.

annb3 commented 6 months ago

@MartyG-RealSense I have SDK 2.55.1 and firmware version 5.16.01 but I have always the problem: [CompressedPublisher]: OpenCV(4.9.0-dev) /home/kitt/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 279965582913600 bytes in function 'OutOfMemoryError' [realsense2_camera_node-33] (publish() at ./src/compressed_publisher.cpp:240)

MartyG-RealSense commented 6 months ago

@annb3 The current ROS wrapper 4.54.1 is not designed for use with SDK 2.55.1 and firmware 5.16.0.1. There is not currently a wrapper for 2.55.1. I note though that @nwissner-bdai has been using 2.54.2 and still also experienced 'OutOfMemoryError'.

Do the compressed topic echoing errors still occur if you use a low resolution and FPS of 640x480 and 15 FPS?

ros2 launch realsense2_camera rs_launch.py depth_module.depth_profile:=640x480x15

nwissner-bdai commented 6 months ago

@MartyG-RealSense So the fw for the realsense camera I was using was not updated to 5.15.1.0, so I updated it and tested again to echo the compressed image topic data. Unfortunately still receiving the same errors. I also added the launch arguments you described and same errors. Do you recommend any other troubleshooting steps before I dig further into the problem?

annb3 commented 6 months ago

@nwissner-bdai I have also 2.55.1 and 5.16.0.1 and ROS wrapper works perfectly- After some experiments I have been able to solve the problem in this way:

open the _rslaunch.py file
modify: 44 {'name': 'depth_module.depth_profile', 'default': '640,480,15', 'description': 'depth stream profile'} ... 71 {'name': 'pointcloud.enable', 'default': 'true', 'description': ''}
save the file
build again the ws
run the launcher

This solution works in my Jetson Orin. Let me know if you will solve in this way or will you find some other solutions..

MartyG-RealSense commented 6 months ago

Thanks so much @annb3 for sharing the difference that enabling the pointcloud filter made for you in your particular situation!

Yes, please do let us know @nwissner-bdai if the above method works for you.

Rebuilding the wrapper may not be a necessary step, as the launch file is independent from the wrapper files and is not affected by a rebuild.

Instead of editing and saving the launch file, the launch command below may produce the same outcome.

ros2 launch realsense2_camera rs_launch.py depth_module.depth_profile:=640x480x15 pointcloud.enable:=true

annb3 commented 6 months ago

@MartyG-RealSense I use the file inside an enourmous repository of concatenate launchers and so on. that's why I can't launch the camera in this way and I have to rebuild (need to modify general settings of the overall project and it is super fast). This is the only possible solution for my purpose.

MartyG-RealSense commented 6 months ago

Thanks very much @annb3 for the clarification about your situation.

nwissner-bdai commented 6 months ago

@MartyG-RealSense Okay after doing some more testing, I have found that adding that additional launch argument you provided still produces the same errors I have been getting. Like @annb3, I too am working with an enormous repository made up of various research projects and am tasked with debugging this specific issue of publishing high resolution compressed image and depth data. My original testing involved launching a node from the repo which also spawned rs_launch.py, but now I have ran the rs_launch.py as a standalone node with and without your launch arguments and am still getting the same console errors when trying to ros2 topic echo /compressionTopicEtc. Specifically:

ros2 topic echo /camera/camera/depth/image_rect_raw/compressed
[realsense2_camera_node-1] [ERROR] [1716407232.444214183] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

and:

ros2 topic echo /camera/camera/color/image_raw/compressed
[realsense2_camera_node-1] [ERROR] [1716407167.873317761] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 118739372698080 bytes in function 'OutOfMemoryError'

I am now going to test this standalone node on non agx-orin hardware (thinkpad) and will update with those results when I can. I'll be sure to double check versioning for the sdk, wrapper and firmware on the camera

MartyG-RealSense commented 6 months ago

Thanks very much @nwissner-bdai for the detailed update. I look forward to your next report after further testing. Good luck!

nwissner-bdai commented 5 months ago

Hello @MartyG-RealSense , I have tested Librealsense SDK versions paired with their appropriate realsense-ros wrapper versions on both the AGX Orin and Lenovo Thinkpad platforms extensively, documenting the topic statistics with ros2 topic hz and ros2 topic bw. I've found that there hasnt been a reliable way to publish compressed image and depth data on the AGX Orin platform for the last 3 major releases, which means this may require some further investigation into the root cause of the issue. I have also collected all the errors that are present when trying to echo the topic data. I will attach all my findings below:

Lenovo Thinkpad

SDK Software Version and Firmware Version

Realsense-ros	4.54.1
Librealsense 2 SDK	2.54.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100 msgs)
/camera/color/image_raw	11.455	29.38
/camera/camera/color/image_raw/compressed	14.987	1.25
/camera/camera/color/image_raw/theora	15.525	22.00
/camera/camera/depth/image_rect_raw	7.041	4.14
/camera/camera/depth/image_rect_raw/compressedDepth	3.524	275.53

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1716919035.770059648] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1716919161.666898386] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1716919219.511761717] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

SDK Software Version and Firmware Version

Realsense-ros	4.51.1
Librealsense 2 SDK	2.51.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw	20.258	52.96
/camera/color/image_raw/compressed	29.992	7.82
/camera/color/image_raw/theora	23.770	7.77
/camera/depth/image_rect_raw	8.766	10.05
/camera/depth/image_rect_raw/compressedDepth	2.204	172.10

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressedDepth

 [realsense2_camera_node-1] [ERROR] [1717440347.780579745] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/depth/image_rect_raw/compressed (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717440501.576116078] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed	14.996	1.27

SDK Software Version and Firmware Version

Realsense-ros	4.55.1
Librealsense 2 SDK	2.55.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw	18.351	56.23
/camera/color/image_raw/compressed	29.976	8.66
/camera/color/image_raw/theora	22.720	15.08
/camera/depth/image_rect_raw	20.233	17.07
/camera/depth/image_rect_raw/compressedDepth	7.295	383.56 (KB/s)

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717446915.676193022] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717447086.328406218] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717447310.293408762] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed	29.986	2.52 (KB/s)
/camera/depth/image_rect_raw/compressedDepth	1.049	212.87 (KB/s)

AGX Orin

SDK Software Version and Firmware Version

Realsense-ros	4.54.1
Librealsense 2 SDK	2.54.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw	29.850	83.70
/camera/depth/image_rect_raw	22.278	19.96
/camera/depth/image_rect_raw/compressedDepth	1.172	214.15 (KB/s)

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717098382.214202425] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 222112060848000 bytes in function 'OutOfMemoryError'

/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717098440.072736737] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/color/image_raw/theora (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717098497.831257488] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 222111994788720 bytes in function 'OutOfMemoryError'

/camera/depth/image_rect_raw/compressed (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717098869.836639329] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/depth/image_rect_raw/compressedDepth (data printing w/ error)

[realsense2_camera_node-1]  30/05 15:56:45,445 ERROR [281472207808736] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 130

/camera/depth/image_rect_raw/theora (data printing w/ error)

[realsense2_camera_node-1] [ERROR] [1717099131.918398786] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/depth/image_rect_raw/compressed	29.977	2.54 (KB/s)
/camera/depth/image_rect_raw/compressedDepth	1.049	212.87 (KB/s)

SDK Software Version and Firmware Version

Realsense-ros	4.51.1
Librealsense 2 SDK	2.51.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/color/image_raw	29.96	66.64
/camera/depth/image_rect_raw	27.993	24.56
/camera/depth/image_rect_raw/compressedDepth	1.158	209.09

Unsuccessful Image and Compression Topics with Error Logs

/camera/color/image_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717201094.335925484] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 180141772297680 bytes in function 'OutOfMemoryError'

/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717201166.633408758] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/color/image_raw/theora

[realsense2_camera_node-1] [ERROR] [1717201199.114697728] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 180141706238400 bytes in function 'OutOfMemoryError'

/camera/depth/image_rect_raw/compressed

[realsense2_camera_node-1] [ERROR] [1717201457.870049961] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717201679.300377455] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

SDK Software Version and Firmware Version

Realsense-ros	4.55.1
Librealsense 2 SDK	2.55.1
D455 Firmware	5.16.0.1

Successful Image and Compression Topics

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/camera/color/image_raw	29.948	73.29
/camera/camera/depth/image_rect_raw	29.959	29.242
/camera/camera/depth/image_rect_raw/compressedDepth	1.041	211.78

Unsuccessful Image and Compression Topics with Error Logs

/camera/camera/color/image_raw/compressed (data printing w/ errors)

[realsense2_camera_node-1] [ERROR] [1717443390.451790943] [CompressedPublisher]: OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 293174478828720 bytes in function 'OutOfMemoryError'

/camera/camera/color/image_raw/compressedDepth

[realsense2_camera_node-1] [ERROR] [1717443507.035844806] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

/camera/camera/color/image_raw/theora

[realsense2_camera_node-1] [ERROR] [1717443535.281818186] [TheoraPublisher]: OpenCV exception: 'OpenCV(4.8.0) /home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 293174412769440 bytes in function 'OutOfMemoryError'

/camera/camera/depth/image_rect_raw/compressed (data printing w/ errors)

[realsense2_camera_node-1] [ERROR] [1717443679.953734063] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

/camera/camera/depth/image_rect_raw/theora

[realsense2_camera_node-1] [ERROR] [1717443926.629635525] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'

Topic Name	ros2 topic hz	ros2 topic bw (MB/s @ 100msgs)
/camera/camera/color/image_raw/compressed	29.923	2.4 (KB/s)
/camera/camera/depth/image_rect_raw/compressed	29.985	2.53 (KB/s)

Apologies if this information doesn't prove to be as useful as I had hoped, but it would be nice if the the community could take a special interest in this issue as it seems to be repeatable and prevalent to other users such as @annb3. If there is any other data you would recommend I collect on this issue I am happy to further investigate!

MartyG-RealSense commented 5 months ago

Thank you very much for the detailed test results. It appears that for both the Lenovo and AGX, /camera/depth/image_rect_raw/compressedDepth is the topic that has the most problems. This may be because the topic is using a very high amount of bandwidth MB compared to the other compressed depth topic /camera/camera/color/image_raw/compressed which has very good performance and small bandwidth usage in comparison.

There was a past case at https://github.com/IntelRealSense/realsense-ros/issues/1672 where a RealSense user also experienced low performance from /camera/depth/image_rect_raw/compressedDepth and they were able to make it faster by changing the value of a setting called png_level.

nwissner-bdai commented 5 months ago

@MartyG-RealSense Thank you for taking such consideration to look into this! I would say /camera/color/image_raw/compressed and /camera/color/image_raw/compressedDepth are the most inconsistent and problematic in the case of the AGX Orin which is the focus of my problem, was only using the Thinkpad as a control. camera/depth/image_rect_raw/compressedDepth at least successfully publishes data in most if not all cases. I did test trying lower values for the png_level parameter but saw marginal difference in bandwidth for the camera/depth/image_rect_raw/compressedDepth topic. Is there any plan on fixing this issue should it occur with other users? I've tried editing the other parameters suggested and am still in no luck of publishing the compressed topics I mentioned earlier

nwissner-bdai commented 5 months ago

@MartyG-RealSense Oh I am working with ROS2 and not ROS1 for context. I do have the ros-humble-image-transport and ros-humble-image-transport-plugins correctly installed

MartyG-RealSense commented 5 months ago

Thank you @nwissner-bdai - I will highlight this issue with high bandwidth usage to my Intel colleagues on the RealSense ROS team.

Edit: the ROS team will attempt to look at this issue at a future date when there is the opportunity to work on it.

annb3 commented 5 months ago

@nwissner-bdai I can confirm that actually somehow I again receive the error: [CompressedPublisher]: OpenCV(4.9.0-dev) /home/kitt/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 279965582913600 bytes in function 'OutOfMemoryError' [realsense2_camera_node-33] (publish() at ./src/compressed_publisher.cpp:240) even if I have already modified the code, so there is something more to be considered. @MartyG-RealSense I suggest to go more deeper testing Release on Orin platform.

nwissner-bdai commented 5 months ago

Hello @MartyG-RealSense , I was just wondering if the ROS team had made any progress on this issue that I could report back to my team as this is a blocking issue for them

MartyG-RealSense commented 4 months ago

Hi @nwissner-bdai I will follow up with the RealSense ROS team about this issue.

MartyG-RealSense commented 4 months ago

Hi @nwissner-bdai My Intel RealSense colleagues have officially created an internal ticket so that they can look at this issue with compression bandwidth at a future date when there is the opportunity to work on it.

nwissner-bdai commented 4 months ago

Sounds good, thank you for getting back to me @MartyG-RealSense

SamerKhshiboun commented 2 months ago

Hi @nwissner-bdai

If you are familiar with building from source (with colcon build), can you please try this fix from my private repo ? https://github.com/SamerKhshiboun/realsense-ros/tree/compressed_images_issue

Let me know if you need help building the code

Thanks, Samer

MartyG-RealSense commented 2 months ago

Hi @nwissner-bdai Have you tried building from the private repo of @SamerKhshiboun using the link in the comment above, please? Thanks!

gbrooks-bdai commented 2 months ago

@MartyG-RealSense I'll be taking over this issue from @nwissner-bdai. It'll take me some time to get onboarded and reproduce but I will test @SamerKhshiboun's branch. I am familiar with building from source but I'll be sure to ask any questions if there are issues :crossed_fingers:

MartyG-RealSense commented 2 months ago

Thanks very much for the information, @gbrooks-bdai - good luck!

edgarcamilocamacho commented 2 months ago

Hello @MartyG-RealSense

I wonder if there is some progress on this issue.

I'm working with two platforms:

LAPTOP
Architecture	x86_64
OS	Ubuntu 22.04.4 LTS
ROS	humble
realsense-ros	4.54.1 (manually built)
librealsense	2.54.1 (manually built)
firmware	5.15.0.2
image_transport	3.1.9 (installed with apt)

JETSON ORIN AGX
Architecture	aarch64
OS	Ubuntu 20.04.6 LTS
Jetpack	5.1.1 [L4T 35.3.1]
ROS	humble
realsense-ros	4.54.1 (manually built)
librealsense	2.54.1 (manually built)
firmware	5.15.0.2
image_transport	3.1.7 (installed with apt)

I'm manually building from the source code (to be able to enable the CUDA and GPU flags in both realsense-ros and librealsense.

I'm not using the last version because the documentation is not clear about the versions I should use (see #3204 and #3205).

I'm getting the error:

[CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

in both platforms, as soon as I subscribe to the depth compressed topic.

Is my combination of versions correct? Or can I install via apt with the GPU flags enabled for both the librealsense and realsense-ros libraries?

Btw @SamerKhshiboun which versions of librealsense, firmware and image-transport should I use to test your branch?

MartyG-RealSense commented 2 months ago

Hi @edgarcamilocamacho If you are using camera firmware driver 5.15.0.2 then librealsense 2.54.1 and ROS wrapper 4.54.1 will be the appropriate versions to use, and you are indeed using those versions.

If you install from packages using the instructions on the installation_jetson.md page then CUDA support will be included in the packages and so will be automatically enabled without you having to set a flag.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

When installing from packages with the distribution_linux.md instructions, CUDA support is not included in the packages. These packages should only be used on x86/x64 computers anyway, whilst the installation_jetson.md packages should be used for Nvidia Jetson.

https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md

If intalling the librealsense SDK from source code, a method that works well for both x86/x64 computers and Arm-based boards such as Jetson is to use the libuvc backend installation method at the link below, which has a pre-made build script called 'libuvc_installation.sh'.

https://github.com/IntelRealSense/librealsense/blob/master/doc/libuvc_installation.md

When using the script with Jetson, edit line 46 to add the flag -DBUILD_WITH_CUDA=ON

https://github.com/IntelRealSense/librealsense/blob/master/scripts/libuvc_installation.sh#L46

In regard to your question about the @SamerKhshiboun repo, I will defer to Samer about that question.

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho Do you require further assistance with your problem, please? Thanks!

SamerKhshiboun commented 1 month ago

Hi,

Please ignore my last proposal for testing the private branch.

I tested the latest RealsSense ROS Wrapper on my Ubuntu 24.04 with ROS2 Jazzy and with my private dynamic_subscriber_node that calculates the HZ (you can use it from here: https://github.com/SamerKhshiboun/dynamic_subscriber_node, please follow the readme with the examples)

I tested these topics:

normal color frame (Image)
compressed color frame (CompressedImage)
normal depth frame (Image)
compressed depth frame (CompressedImage)

The results on all of them are 30.0 FPS.

I think using a custom C++ subscriber is better than using the ros2 topic hz, you can search and read about this online, but I think the QoS of the ros2 topic hz is making the pub/sub result slower. In addition to that, the ROS2 HZ was implemented with rclpy (the python API) which also make it slower. Some references: https://answers.ros.org/question/350753/ros2-topic-hz-provides-wrong-rate-for-larger-msgs/ https://github.com/ros2/ros2/issues/1499 https://github.com/ros2/ros2cli/issues/871

> ros2 run dynamic_subscriber_node dynamic_subscriber_node --ros-args -p topic_name:=/camera/camera/color/image_raw -p topic_type:=sensor_msgs/msg/Image
[INFO] [1727993085.868889460] [dynamic_subscriber_node]: Subscribing to sensor_msgs/msg/Image on topic: /camera/camera/color/image_raw
[INFO] [1727993086.869076958] [dynamic_subscriber_node]: Current Frequency (Hz): 30.05

> ros2 run dynamic_subscriber_node dynamic_subscriber_node --ros-args -p topic_name:=/camera/camera/color/image_raw/compressed -p topic_type:=sensor_msgs/msg/CompressedImage
[INFO] [1727993057.415358008] [dynamic_subscriber_node]: Subscribing to sensor_msgs/msg/CompressedImage on topic: /camera/camera/color/image_raw/compressed
[INFO] [1727993058.415583860] [dynamic_subscriber_node]: Current Frequency (Hz): 30.82

> ros2 run dynamic_subscriber_node dynamic_subscriber_node --ros-args -p topic_name:=/camera/camera/depth/image_rect_raw -p topic_type:=sensor_msgs/msg/Image
[INFO] [1727993150.739524810] [dynamic_subscriber_node]: Subscribing to sensor_msgs/msg/Image on topic: /camera/camera/depth/image_rect_raw
[INFO] [1727993151.739712993] [dynamic_subscriber_node]: Current Frequency (Hz): 29.92

> ros2 run dynamic_subscriber_node dynamic_subscriber_node --ros-args -p topic_name:=/camera/camera/depth/image_rect_raw/compressedDepth -p topic_type:=sensor_msgs/msg/CompressedImage
[INFO] [1727993178.346787800] [dynamic_subscriber_node]: Subscribing to sensor_msgs/msg/CompressedImage on topic: /camera/camera/depth/image_rect_raw/compressedDepth
[INFO] [1727993179.346900218] [dynamic_subscriber_node]: Current Frequency (Hz): 29.49

Please retry again with my custom C++ subscriber, or create your own, and update me.

edgarcamilocamacho commented 1 month ago

Amazing, we'll be testing in the next days and comment how it goes.

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please? Thanks!

tiwaojo commented 1 month ago

Hi @edgarcamilocamacho If you are using camera firmware driver 5.15.0.2 then librealsense 2.54.1 and ROS wrapper 4.54.1 will be the appropriate versions to use, and you are indeed using those versions.

If you install from packages using the instructions on the installation_jetson.md page then CUDA support will be included in the packages and so will be automatically enabled without you having to set a flag.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

When installing from packages with the distribution_linux.md instructions, CUDA support is not included in the packages. These packages should only be used on x86/x64 computers anyway, whilst the installation_jetson.md packages should be used for Nvidia Jetson.

https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md

If intalling the librealsense SDK from source code, a method that works well for both x86/x64 computers and Arm-based boards such as Jetson is to use the libuvc backend installation method at the link below, which has a pre-made build script called 'libuvc_installation.sh'.

https://github.com/IntelRealSense/librealsense/blob/master/doc/libuvc_installation.md

When using the script with Jetson, edit line 46 to add the flag -DBUILD_WITH_CUDA=ON

https://github.com/IntelRealSense/librealsense/blob/master/scripts/libuvc_installation.sh#L46

In regard to your question about the @SamerKhshiboun repo, I will defer to Samer about that question.

Hello @MartyG-RealSense, the proposed method above doesn't seem to work. The steps i followed are as follows:

Clone the librealsense repository git clone https://github.com/IntelRealSense/librealsense.git
Change directory to librealsense cd librealsense
install the dependencies sudo apt-get install libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev ninja-build -y
Setup udev rules ./scripts/setup_udev_rules.sh && sudo udevadm control --reload-rules && sudo udevadm trigger
Configure cmake -S . -B build -G Ninja -DFORCE_LIBUVC=true -DBUILD_EXAMPLES=true -DCMAKE_BUILD_TYPE=release -DBUILD_WITH_CUDA=true -DFORCE_RSUSB_BACKEND=false -DBUILD_GRAPHICAL_EXAMPLES=ON
Build and install cmake --build build && sudo cmake --install build
Reconnect the camera
launch the realsense2_camera node ros2 launch realsense2_camera rs_launch.py
I got similar results as @SamerKhshiboun with his dynamic_subscriber_node, though I wasn't able to get any messages on the /camera/camera/color/image_raw/ffmpeg while using the ffmpeg_image_transport. i.e. ros2 run dynamic_subscriber_node dynamic_subscriber_node --ros-args -p topic_name:=/camera/camera/color/image_raw/ffmpeg -p topic_type:=ffmpeg_image_transport_msgs/msg/FFMPEGPacket

I also verified depth and pointcloud topics were working. i.e. /camera/camera/depth/image_rect_raw and /camera/camera/depth/color/points. My param configuration is as follows:

enable_color: true
enable_depth: true
enable_sync: true
rgb_camera.color_profile: 1920x1080x30 # results are the same with 640x480x30
rgb_camera.color_format: RGB8
publish_tf: true
initial_reset: true
pointcloud.enable: true

Moreover, after launching the realsense2_camera node, I got the following warning:

[realsense2_camera_node-1]  25/10 15:47:31,851 WARNING [281472055044320] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1]  25/10 15:47:32,167 WARNING [281472055044320] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1]  25/10 15:47:32,178 WARNING [281472055044320] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

My realsense environment is as follows:

Device Name: Intel RealSense D435
Built with LibRealSense v2.55.1
RealSense ROS v4.55.1

I appreciate your assistance in this matter as well as tips I could try. Thank you.

MartyG-RealSense commented 4 weeks ago

Hi @tiwaojo Which ROS2 branch are you using, please? The ROS2 repository of ffmpeg_image_transport says that it has been tested with ROS2 Humble and Iron.

https://github.com/ros-misc-utilities/ffmpeg_image_transport

Have you confirmed that /camera/camera/color/image_raw/ffmpeg is in the list of topics being published by using ros2 topic list

tiwaojo commented 4 weeks ago

My apologies, I was using ROS2 Humble and yes, the /camera/camera/color/image_raw/ffmpeg is present in my list of topics but no messages were published on it. The 'OutOfMemory' error message as mentioned by the issue poster is printed to stdout if a subscription is made on that topic.

MartyG-RealSense commented 4 weeks ago

Thank you for the information. Are ffmpeg messages able to be published if the pointcloud filter is not enabled?

tiwaojo commented 3 weeks ago

Unfortunately no, disabling the pointcloud does not result in ffmpeg messages being published.

MartyG-RealSense commented 3 weeks ago

@tiwaojo I note that you have used the flag -DBUILD_WITH_CUDA=true when installing librealsense. Can you confirm if you are using an Nvidia Jetson board as your computing device, please?

AIf you are using a Jetson, because you get an OutOfMemory error when subscribing to the topic, have you tried reducing the load on the CPU by encoding in hevc_nvenc format using the ffmpeg_image_transport plugin's instructions at the link below?

https://github.com/ros-misc-utilities/ffmpeg_image_transport?tab=readme-ov-file#publisher-camera-driver

tiwaojo commented 3 weeks ago

Pardon the delay, we ran into some issues with setting up ffmpeg on the AGX and were troubleshooting.

[realsense2_camera_node-1] [ERROR] [1730172697.698395765] [FFMPEGEncoder]: cannot find encoder: h264_nvmpi

We followed steps as documented here with supporting documents from nvidia and online to solve the problem.

We had some success while using the ros2_v4l2_camera package with the ffmpeg transport by manually specifying the /dev/video# camera to use. Regardless, we still encounter the error with the realsense2_camera node.

Moreover, we tried to play back a bag file to compress and decompress the messages using the ffmpeg_image_transport and image transport, but could not replicate the OutOfMemoryError issue.

The opencv version i have on my jetson device is 4.5.4.

MartyG-RealSense commented 3 weeks ago

AS you are using a Jetson, can you confirm if you followed the instructions to install hardware accelerated encoding in ffmpeg_image transport, please?

https://github.com/ros-misc-utilities/ffmpeg_image_transport?tab=readme-ov-file#how-to-use-ffmpeg-hardware-accelerated-encoding-on-the-nvidia-jetson

tiwaojo commented 3 weeks ago

AS you are using a Jetson, can you confirm if you followed the instructions to install hardware accelerated encoding in ffmpeg_image transport, please?

https://github.com/ros-misc-utilities/ffmpeg_image_transport?tab=readme-ov-file#how-to-use-ffmpeg-hardware-accelerated-encoding-on-the-nvidia-jetson

Yes, we followed the instructions in the link you shared to install the h264_nvmpi encoder. We confirmed the encoder works with the ffmpeg_image_transport by using the ros2_v4l2_camera package to capture, compress(via ffmpeg_image_tranport) and stream frames from my Intel RealSense D435 by specifing the /dev/video path to the camera.

MartyG-RealSense commented 3 weeks ago

And reducing the resolution of the color stream makes no difference?

ros2 launch realsense2_camera rs_launch.py rgb_camera.color_profile:=640x480x30

agonzat commented 3 weeks ago

Hello, I can confirm that we can reproduce these issues under Jetpack 6.0 on a Jetson Orin AGX board with ROS 2 Humble.

I have librealsense 2.55.1 and ROS 2 wrapper realsense-ros 4.5.1 with cameras on the 5.16.0.1 firmware. (Also on OpenCV 4.8).

librealsense is installed following the libuvc_installation.md script with CUDA enabled.

This setup worked previously, properly receiving compressed images from the compressed topic, as all of these components (librealsense and realsense-ros) are built from source and have remained unchanged. The issue only popped up after upgrading the apt pakcages/dependencies and rebuilding the wrapper.

Which produces CompressedImage messages with empty payloads, and logs an error on the realsense2_camera_node, when a subscription to the topic is active:

[realsense2_camera_node-1] [ERROR] [1730302396.252741557] [CompressedPublisher]: OpenCV(4.8.0)
/home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 
36307410248880 bytes in function 'OutOfMemoryError'

All of this is consistent with what other users have posted here.

We haven't been using the ffmpeg transport, although giving a try to the instructions referenced by @MartyG-RealSense didn't work for us; instead, we have been using the compressed topic on the RGB channel (relying just on image_transport_plugins). The versions we currently have are:

ii  ros-humble-image-transport                                        3.1.9-1jammy.20240820.161232                arm64        image_transport should always be used to subscribe to and publish images.
ii  ros-humble-image-transport-plugins                                2.5.2-1jammy.20240820.173314                arm64        A set of plugins for publishing and subscribing to sensor_msgs/Image topics in representations other than raw pixel data.

This issue is consistent regardless of the profile selected for the color channel in our case.

tiwaojo commented 3 weeks ago

And reducing the resolution of the color stream makes no difference?

ros2 launch realsense2_camera rs_launch.py rgb_camera.color_profile:=640x480x30

Thats Correct.

MartyG-RealSense commented 3 weeks ago

https://github.com/IntelRealSense/realsense-ros/issues/2858 is a previously reported case from 2023 of problems with using ffmpeg with RealSense ROS.

At https://github.com/IntelRealSense/realsense-ros/issues/2858#issuecomment-1695448502 I suggested to the RealSense user in that case to try using the -map function of ffmpeg.

Later in that case at https://github.com/IntelRealSense/realsense-ros/issues/2858#issuecomment-1697323625 my Intel RealSense colleagues affirmed that the advice to use the map function was a suitable answer and also suggested an alternative solution.

tiwaojo commented 3 weeks ago

Thank you. We will consider both options and report back with our findings once we I get the chance to.

agonzat commented 2 weeks ago

I believe to have found a workaround for:

[realsense2_camera_node-1] [ERROR] [1730302396.252741557] [CompressedPublisher]: OpenCV(4.8.0)
/home/ubuntu/opencv_build/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 
36307410248880 bytes in function 'OutOfMemoryError'

cv_bridge attempts conversion to bgr8 format on the rgb images prior to conversion and in step it crashes. If the camera driver's format is changed to match these same encoding ({"rgb_camera.color_format": "BGR8"}) the copying doesn't happen and instead the message's buffer is reused for compression avoiding/circumventing the error.

This seems to point to some issue with either cv:cvtColor (used internally) or with how cv_bridge handles copying of images on Jetson platforms. Also cv_bridge on Humble is meant to build against OpenCV 4.54.1 (as available on the distribution for Ubuntu 22.04 x64) and on Jetpack 6.0 the OpenCV version is bumped to 4.8.0, although it shouldn't be a problem (API/semantics changes, etc.) in such low level operations.

TL;DR; Set the RGB stream's encoding to BGR8 ("rgb_camera.color_format": "BGR8") to get the compressed image topic to work.

MartyG-RealSense commented 2 weeks ago

Thanks so much @agonzat for sharing your knowledge of what worked for you!

edgarcamilocamacho commented 1 week ago

I think the "BGR8" solution does not apply for the original issue, because it was about depth.

I made the change just in clase and I still get:

[nav_bottom-5] [ERROR] [1731349335.302148450] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense

When I try to subscribe to the compressed depth

IntelRealSense / realsense-ros

Failing to publish compressed image and depth topic data on Nvidia AGX Orin #3105

Issue Description

Lenovo Thinkpad

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs

AGX Orin

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs

SDK Software Version and Firmware Version

Successful Image and Compression Topics

Unsuccessful Image and Compression Topics with Error Logs