search

IntelRealSense / realsense-ros

ROS Wrapper for Intel(R) RealSense(TM) Cameras
http://wiki.ros.org/RealSense
Apache License 2.0
2.55k stars 1.75k forks source link

When I turn on the depth camera and the imu at the same time, an error occurs #2794

Closed SenHe524 closed 1 year ago

SenHe524 commented 1 year ago

When I tried to turn on the depth camera and the imu at the same time, I got an error with the following message:

[realsense2_camera_node-1] [INFO] [1688006264.529313899] [camera.camera]: Set ROS param rgb_camera.profile to default: 1280x720x30 [realsense2_camera_node-1] [INFO] [1688006264.535419429] [camera.camera]: Set ROS param gyro_fps to default: 200 [realsense2_camera_node-1] [INFO] [1688006264.535830725] [camera.camera]: Set ROS param accel_fps to default: 63 [realsense2_camera_node-1] [INFO] [1688006264.602810662] [camera.camera]: Stopping Sensor: Depth Module [realsense2_camera_node-1] [INFO] [1688006264.624322610] [camera.camera]: Starting Sensor: Depth Module [realsense2_camera_node-1] [INFO] [1688006264.659667697] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 848, Height: 480, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006264.659862641] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 848, Height: 480, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006264.660711024] [camera.camera]: Stopping Sensor: RGB Camera [realsense2_camera_node-1] [INFO] [1688006264.678453056] [camera.camera]: Starting Sensor: RGB Camera [realsense2_camera_node-1] [INFO] [1688006264.692967698] [camera.camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 1280, Height: 720, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006264.693287442] [camera.camera]: Stopping Sensor: Motion Module [realsense2_camera_node-1] 28/06 19:37:44,697 WARNING [281473256024480] (ds-calib-parsers.cpp:41) IMU Calibration is not available, default intrinsic and extrinsic will be used. [realsense2_camera_node-1] [INFO] [1688006264.706518406] [camera.camera]: Starting Sensor: Motion Module [realsense2_camera_node-1] [INFO] [1688006264.719790585] [camera.camera]: Open profile: stream_type: Accel(0)Format: MOTION_XYZ32F, FPS: 63 [realsense2_camera_node-1] [INFO] [1688006264.719968409] [camera.camera]: Open profile: stream_type: Gyro(0)Format: MOTION_XYZ32F, FPS: 200 [realsense2_camera_node-1] [INFO] [1688006264.728618577] [camera.camera]: RealSense Node Is Up! [realsense2_camera_node-1] [WARN] [1688006264.847343618] [camera.camera]: [realsense2_camera_node-1] 28/06 19:38:18,030 ERROR [281472543967648] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources! [realsense2_camera_node-1] 28/06 19:38:18,062 ERROR [281472543967648] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources! [realsense2_camera_node-1] 28/06 19:38:18,096 ERROR [281472543967648] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources! [realsense2_camera_node-1] 28/06 19:38:18,129 ERROR [281472543967648] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources! [realsense2_camera_node-1] 28/06 19:38:18,162 ERROR [281472543967648] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources!

Sometimes the error is as follows:

[realsense2_camera_node-1] [INFO] [1688006905.246600832] [camera.camera]: Set ROS param gyro_fps to default: 200 [realsense2_camera_node-1] [INFO] [1688006905.247139364] [camera.camera]: Set ROS param accel_fps to default: 63 [realsense2_camera_node-1] [INFO] [1688006905.315500302] [camera.camera]: Stopping Sensor: Depth Module [realsense2_camera_node-1] [INFO] [1688006905.339641617] [camera.camera]: Starting Sensor: Depth Module [realsense2_camera_node-1] [INFO] [1688006905.376287130] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 848, Height: 480, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006905.376468955] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 848, Height: 480, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006905.377003455] [camera.camera]: Stopping Sensor: RGB Camera [realsense2_camera_node-1] [INFO] [1688006905.389023297] [camera.camera]: Starting Sensor: RGB Camera [realsense2_camera_node-1] [INFO] [1688006905.403627479] [camera.camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 1280, Height: 720, FPS: 30 [realsense2_camera_node-1] [INFO] [1688006905.403934521] [camera.camera]: Stopping Sensor: Motion Module [realsense2_camera_node-1] 28/06 19:48:25,408 WARNING [281472023914912] (ds-calib-parsers.cpp:41) IMU Calibration is not available, default intrinsic and extrinsic will be used. [realsense2_camera_node-1] [INFO] [1688006905.418896498] [camera.camera]: Starting Sensor: Motion Module [realsense2_camera_node-1] [INFO] [1688006905.431899004] [camera.camera]: Open profile: stream_type: Accel(0)Format: MOTION_XYZ32F, FPS: 63 [realsense2_camera_node-1] [INFO] [1688006905.432195710] [camera.camera]: Open profile: stream_type: Gyro(0)Format: MOTION_XYZ32F, FPS: 200 [realsense2_camera_node-1] [INFO] [1688006905.446813012] [camera.camera]: RealSense Node Is Up! [realsense2_camera_node-1] [WARN] [1688006905.559998473] [camera.camera]: [realsense2_camera_node-1] [WARN] [1688006906.341799318] [camera.camera]: XXX Hardware Notification:Left MIPI error,1.68801e+12,Error,Hardware Error [realsense2_camera_node-1] [WARN] [1688006906.342063640] [camera.camera]: Hardware Notification:Left MIPI error,1.68801e+12,Error,Hardware Error

If I had just turned on the depth camera, or just turned on the imu, none of these errors would have happened. My development environment is: jetson xavier nx, ubuntu20.04,foxy.

MartyG-RealSense commented 1 year ago

Hi @SenHe524 Does the hardware error still occur if you reset the camera at launch with initial_reset:=true

ros2 run ros2 run realsense2_camera realsense2_camera_node --ros-args -p enable_gyro:=true -p enable_accel:=true -p unite_imu_method:=2 -p initial_reset:=true

ros2 launch ros2 launch realsense2_camera rs_launch.py enable_gyro:=true enable_accel:=true unite_imu_method:=2 initial_reset:=true

SenHe524 commented 1 year ago

Yes, I tried to reset the camera by setting initial_reset:=true at startup, but the error still persisted. But in subsequent attempts, sometimes the imu and the depth camera worked together, and sometimes they didn't. Sometimes, both of the above errors occur at the same time, as shown in the figure below: image

MartyG-RealSense commented 1 year ago

Which model of RealSense camera are you using, please?

SenHe524 commented 1 year ago

The environment in which the above error occurs is： jetson xavier nx, ubuntu20.04,foxy，D455

MartyG-RealSense commented 1 year ago

And the problem still occurs if you disable the color stream with enable_color:=false ?

SenHe524 commented 1 year ago

Yes, it still happens. But yesterday I tried to reinstall Intel® RealSense™ SDK 2.0 using the installation method shown in the link, and the imu and Depth camera can be used together. But errors are still reported: image

MartyG-RealSense commented 1 year ago

When a Motion Module failure occurs in the RealSense Viewer then it usually works after the Motion Module (IMU) is enabled a second time afterwards, indicating that it is a software glitch rather than a failure of the IMU hardware. In the ROS wrapper, the only way to enable the Motion Module a second time after this warning appears would be to exit out of the launch with CTRL-C and then repeat the launch instruction.

SenHe524 commented 1 year ago

Well. I also want to ask a question about imu, that is, why the quaternion field of imu topic is all zero?

MartyG-RealSense commented 1 year ago

Another RealSense ROS user at https://github.com/IntelRealSense/realsense-ros/issues/2241#issuecomment-1029821342 also had all-zero IMU quaternions. I suggested to them to obtain the IMU quaternions using a ROS package called imu_tools like a user at https://github.com/IntelRealSense/realsense-ros/issues/1932#issuecomment-864157132 did.

SenHe524 commented 1 year ago

So you're saying that it's normal for the imu's quaternions to be all zero, not a hardware or software error?

Another RealSense ROS user at #2241 (comment) also had all-zero IMU quaternions. I suggested to them to obtain the IMU quaternions using a ROS package called imu_tools like a user at #1932 (comment) did.

MartyG-RealSense commented 1 year ago

I don't personally know if it is normal for them all to be zero because I am only aware of the cases at https://github.com/IntelRealSense/realsense-ros/issues/1932#issuecomment-864157132 and https://github.com/IntelRealSense/realsense-ros/issues/2241 where people have tried viewing the quaternions.

SenHe524 commented 1 year ago

I don't personally know if it is normal for them all to be zero because I am only aware of the cases at #1932 (comment) and #2241 where people have tried viewing the quaternions.

You mean that it is not normal for imu to have all zero quaternions, but there is no good solution so far, only the solution you mentioned above, right? ps: I use translation software to communicate with you, if the translation is incomplete or impolite, please forgive me!

MartyG-RealSense commented 1 year ago

Do not worry, it is no trouble at all :)

I do not know if it is normal. I just know that the recommendation to use imu_tools to get the quaternions seemed to solve the problem for the user at https://github.com/IntelRealSense/realsense-ros/issues/2241

SenHe524 commented 1 year ago

Do not worry, it is no trouble at all :)

I do not know if it is normal. I just know that the recommendation to use imu_tools to get the quaternions seemed to solve the problem for the user at #2241

Ok, thank you for your patience.

MartyG-RealSense commented 1 year ago

Hi @SenHe524 Do you require further assistance with this case, please? Thanks!

SenHe524 commented 1 year ago

Hi @SenHe524 Do you require further assistance with this case, please? Thanks! Hi @MartyG-RealSense，Of the problems mentioned above, only the inability to obtain imu and depth camera data at the same time has been solved, and other problems still exist, such as： image. image

I haven't tried to get the quaternion of the imu using other methods, so I will try this later.Thanks!

MartyG-RealSense commented 1 year ago

The Motion Module Failure warning is not predictable or preventable. If it occurs, the only solution is to re-enable the IMU stream (or perform launch again, in the case of the ROS wrapper).

The "Exception was thrown during callback: Out of frame resouces!" message could be related to a lack of system resources, where the computer / computing device cannot keep up with the incoming flow of images. Reducing frame rate (FPS) and / or resolution if possible to do so may help to deal with this.

When using an Nvidia Jetson board, having CUDA support enabled in the librealsense SDK if it is not already enabled can improve the situation, as it can offload the processing work of depth-color alignment, pointclouds and RGB color conversion from the CPU onto the Jetson's graphics GPU. CUDA support is automatically enabled if librealsense is built from the SDK's Jetson version of packages, or can be added manually when building librealsense from source code by including the flag -DBUILD_WITH_CUDA=ON in the CMake build instruction.

SenHe524 commented 1 year ago

Thank you for your reply, I will try the solution you said in the future, thank you!

The Motion Module Failure warning is not predictable or preventable. If it occurs, the only solution is to re-enable the IMU stream (or perform launch again, in the case of the ROS wrapper).

The "Exception was thrown during callback: Out of frame resouces!" message could be related to a lack of system resources, where the computer / computing device cannot keep up with the incoming flow of images. Reducing frame rate (FPS) and / or resolution if possible to do so may help to deal with this.

When using an Nvidia Jetson board, having CUDA support enabled in the librealsense SDK if it is not already enabled can improve the situation, as it can offload the processing work of depth-color alignment, pointclouds and RGB color conversion from the CPU onto the Jetson's graphics GPU. CUDA support is automatically enabled if librealsense is built from the SDK's Jetson version of packages, or can be added manually when building librealsense from source code by including the flag -DBUILD_WITH_CUDA=ON in the CMake build instruction.

MartyG-RealSense commented 1 year ago

Hi @SenHe524 Do you require further assistance with this case, please? Thanks!

SenHe524 commented 1 year ago

Recently, I have been busy with other things, and I am sorry that I have not verified the method you proposed above.

Hi @SenHe524 Do you require further assistance with this case, please? Thanks!

MartyG-RealSense commented 1 year ago

Okay, thanks very much for the update. I will keep this case open for a further time period.

MartyG-RealSense commented 1 year ago

Hi @SenHe524 Do you have an update about this case that you can provide, please? Thanks!

SenHe524 commented 1 year ago

Hi @SenHe524 Do you have an update about this case that you can provide, please? Thanks! Hi @MartyG-RealSense When I tried to use the D455 camera recently, I still got the error mentioned above, but the camera can capture the picture normally, and I have not solved the problem of no quaternion in the imu. That is my current situation with the camera, thank you for your continued attention. I think maybe you can close this issue, I will not have much update for a while

MartyG-RealSense commented 1 year ago

Thanks very much @SenHe524 for the update. As you suggest, I will close the issue and you are welcome to re-open it at a future date if you need to. Thanks again!

SenHe524 commented 1 year ago

Thanks very much @SenHe524 for the update. As you suggest, I will close the issue and you are welcome to re-open it at a future date if you need to. Thanks again!

Thank you for your support @MartyG-RealSense