D435i imu data lossing

[zhizunhu]

I have two realsense d435i. When I open the dirver with imu interpolation on, I found that both of them have imu data lost Hear is my config yaml: And I record a rosbag:

[MartyG-RealSense]

Hi @zhizunhu Do you still experience IMU data loss if unite_imu_method is not used?

Also, could you confirm please the extent of the missing data. Is it dropped frames, or no IMU data at all?

[zhizunhu]

Hi @MartyG-RealSense Yes, I close unite_imu_method, It still has loss: And the freq of gyro is not consistent like that (the mid is the sample of gyro)： And I found that there is no loss in visual frames.

[MartyG-RealSense]

Thank you very much for the charts. So the large white gaps in the gyro image represent missing frames that should be there?

[zhizunhu]

Yes @MartyG-RealSense , each blue line means one sensor sample. They are placed by timestamp. Big white gaps means small freq. The imu datas of D435i I tested were not sampled by a consistent freq. Then I just play the rosbag recorded and echo the imu topic, found that there is some loss like that: And it always lose 3 samples. But the seq was added, Does it mean that the driver has solved these data but not published?

[MartyG-RealSense]

Does adding the flag initial_reset:=true to your roslaunch statement to perform a hardware reset of the camera at launch improve the results, please?

Regarding your final comment beneath the image: I do not have sufficient knowledge of that particular subject to comment on it unfortunately.

[skiddattle]

I think I may have a similar issue, where the /camer/imu topic freezes every now and then. I am using ubuntu 18.04 on a laptop.

Here is a youtube link to the freezing output: https://youtu.be/PkUiWxzqiK4

I have tried initial_reset:=true and it doesn't improve performance. I have also checked to accel and gyro topics to see if there is any freezing and it doesnt show any freezing.

For unite imu method I used linear_interpolation.

It will update, and freeze at a certain point, and then update again after a while, and this cycle repeats.

Command:

rostopic echo /camera/imu

Console output:

header: 
  seq: 25295
  stamp: 
    secs: 1604058751
    nsecs: 334259987
  frame_id: "camera_imu_optical_frame"
orientation: 
  x: 0.0
  y: 0.0
  z: 0.0
  w: 0.0
orientation_covariance: [-1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
angular_velocity: 
  x: 8.73358385434e-06
  y: -0.00338855828159
  z: -0.00173951964825
angular_velocity_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01]
linear_acceleration: 
  x: -0.0595852163378
  y: -9.80048897499
  z: 0.289907492512
linear_acceleration_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01]

Command:

roslaunch realsense2_camera rs_camera.launch

Console output:

bryan@bryan-ThinkPad-T450:~$ roslaunch realsense2_camera rs_camera.launch
... logging to /home/bryan/.ros/log/cc1f7d4e-1aa5-11eb-893b-94659c228819/roslaunch-bryan-ThinkPad-T450-6104.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://bryan-ThinkPad-T450:33423/

SUMMARY
========

PARAMETERS
 * /camera/realsense2_camera/accel_fps: 250
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: False
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/allow_no_texture_points: False
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: -2.0
 * /camera/realsense2_camera/color_fps: 6
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 480
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 640
 * /camera/realsense2_camera/depth_fps: 6
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 480
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 640
 * /camera/realsense2_camera/device_type: 
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: False
 * /camera/realsense2_camera/enable_depth: False
 * /camera/realsense2_camera/enable_fisheye1: True
 * /camera/realsense2_camera/enable_fisheye2: True
 * /camera/realsense2_camera/enable_fisheye: False
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: True
 * /camera/realsense2_camera/enable_infra2: True
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_sync: False
 * /camera/realsense2_camera/filters: 
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: 6
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: 480
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: 640
 * /camera/realsense2_camera/gyro_fps: 400
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/imu_optical_frame_id: camera_imu_optica...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 15
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_width: 640
 * /camera/realsense2_camera/initial_reset: False
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: True
 * /camera/realsense2_camera/publish_tf: True
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/tf_publish_rate: 0.0
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: linear_interpolation
 * /camera/realsense2_camera/usb_port_id: 
 * /rosdistro: melodic
 * /rosversion: 1.14.9

NODES
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

ROS_MASTER_URI=http://localhost:11311

process[camera/realsense2_camera_manager-1]: started with pid [6119]
process[camera/realsense2_camera-2]: started with pid [6120]
[ INFO] [1604058526.993961037]: Initializing nodelet with 4 worker threads.
[ INFO] [1604058527.090390544]: RealSense ROS v2.2.14
[ INFO] [1604058527.090441791]: Running with LibRealSense v2.35.2
[ INFO] [1604058527.099380174]:  
[ INFO] [1604058527.208432873]: Device with serial number 843112071857 was found.

[ INFO] [1604058527.208560226]: Device with physical ID /sys/devices/pci0000:00/0000:00:14.0/usb3/3-1/3-1:1.0/video4linux/video2 was found.
[ INFO] [1604058527.208635318]: Device with name Intel RealSense D435I was found.
[ INFO] [1604058527.209766180]: Device with port number 3-1 was found.
[ INFO] [1604058527.214349240]: getParameters...
[ INFO] [1604058527.269346525]: setupDevice...
[ INFO] [1604058527.269392207]: JSON file is not provided
[ INFO] [1604058527.269414315]: ROS Node Namespace: camera
[ INFO] [1604058527.269442448]: Device Name: Intel RealSense D435I
[ INFO] [1604058527.269470633]: Device Serial No: 843112071857
[ INFO] [1604058527.269519245]: Device physical port: /sys/devices/pci0000:00/0000:00:14.0/usb3/3-1/3-1:1.0/video4linux/video2
[ INFO] [1604058527.269542058]: Device FW version: 05.12.07.100
[ INFO] [1604058527.269561191]: Device Product ID: 0x0B3A
[ INFO] [1604058527.269578901]: Enable PointCloud: Off
[ INFO] [1604058527.269597589]: Align Depth: Off
[ INFO] [1604058527.269613179]: Sync Mode: Off
[ INFO] [1604058527.269661723]: Device Sensors: 
[ INFO] [1604058527.271641254]: Stereo Module was found.
 30/10 19:48:47,271 WARNING [140135210444544] (backend-v4l2.cpp:1223) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
[ INFO] [1604058527.275733109]: RGB Camera was found.
[ INFO] [1604058527.275868895]: Motion Module was found.
[ INFO] [1604058527.275910897]: (Infrared, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1604058527.275954175]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[ INFO] [1604058527.275988440]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[ INFO] [1604058527.276013742]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1604058527.276049990]: num_filters: 0
[ INFO] [1604058527.276068260]: Setting Dynamic reconfig parameters.
[ INFO] [1604058527.366470085]: Done Setting Dynamic reconfig parameters.
[ INFO] [1604058527.371140944]: infra1 stream is enabled - width: 640, height: 480, fps: 15, Format: Y8
[ INFO] [1604058527.371381238]: infra2 stream is enabled - width: 640, height: 480, fps: 15, Format: Y8
[ INFO] [1604058527.371446918]: setupPublishers...
[ INFO] [1604058527.373800272]: Expected frequency for infra1 = 15.00000
[ INFO] [1604058527.405372518]: Expected frequency for infra2 = 15.00000
[ INFO] [1604058527.427135432]: Start publisher IMU
[ INFO] [1604058527.427922348]: setupStreams...
[ INFO] [1604058527.428400649]: insert Infrared to Stereo Module
[ INFO] [1604058527.428470137]: insert Infrared to Stereo Module
[ INFO] [1604058527.428518534]: insert Gyro to Motion Module
[ INFO] [1604058527.428563242]: insert Accel to Motion Module
[ INFO] [1604058527.433802409]: SELECTED BASE:Depth, 0
[ INFO] [1604058527.449940385]: RealSense Node Is Up!
 30/10 19:48:47,562 WARNING [140134901077760] (sensor.cpp:970) HID timestamp not found, switching to Host timestamps.

rs_camera.launch file:

File contents:

<launch>
  <arg name="serial_no"           default=""/>
  <arg name="usb_port_id"         default=""/>
  <arg name="device_type"         default=""/>
  <arg name="json_file_path"      default=""/>
  <arg name="camera"              default="camera"/>
  <arg name="tf_prefix"           default="$(arg camera)"/>
  <arg name="external_manager"    default="false"/>
  <arg name="manager"             default="realsense2_camera_manager"/>

  <arg name="fisheye_width"       default="640"/>
  <arg name="fisheye_height"      default="480"/>
  <arg name="enable_fisheye"      default="false"/>

  <arg name="depth_width"         default="640"/>
  <arg name="depth_height"        default="480"/>
  <arg name="enable_depth"        default="false"/>

  <arg name="infra_width"        default="640"/>
  <arg name="infra_height"       default="480"/>
  <arg name="enable_infra"        default="false"/>
  <arg name="enable_infra1"       default="true"/>
  <arg name="enable_infra2"       default="true"/>

  <arg name="color_width"         default="640"/>
  <arg name="color_height"        default="480"/>
  <arg name="enable_color"        default="false"/>

  <arg name="fisheye_fps"         default="30"/>
  <arg name="depth_fps"           default="30"/>
  <arg name="infra_fps"           default="15"/>
  <arg name="color_fps"           default="30"/>
  <arg name="gyro_fps"            default="400"/>
  <arg name="accel_fps"           default="250"/>
  <arg name="enable_gyro"         default="true"/>
  <arg name="enable_accel"        default="true"/>

  <arg name="enable_pointcloud"         default="false"/>
  <arg name="pointcloud_texture_stream" default="RS2_STREAM_COLOR"/>
  <arg name="pointcloud_texture_index"  default="0"/>

  <arg name="enable_sync"               default="false"/>
  <arg name="align_depth"               default="false"/>

  <arg name="publish_tf"                default="true"/>
  <arg name="tf_publish_rate"           default="0"/>

  <arg name="filters"                   default=""/>
  <arg name="clip_distance"             default="-2"/>
  <arg name="linear_accel_cov"          default="0.01"/>
  <arg name="initial_reset"             default="false"/>
  <arg name="unite_imu_method"          default="linear_interpolation"/>
  <arg name="topic_odom_in"             default="odom_in"/>
  <arg name="calib_odom_file"           default=""/>
  <arg name="publish_odom_tf"           default="true"/>
  <arg name="allow_no_texture_points"   default="false"/>

  <group ns="$(arg camera)">
    <include file="$(find realsense2_camera)/launch/includes/nodelet.launch.xml">
      <arg name="tf_prefix"                value="$(arg tf_prefix)"/>
      <arg name="external_manager"         value="$(arg external_manager)"/>
      <arg name="manager"                  value="$(arg manager)"/>
      <arg name="serial_no"                value="$(arg serial_no)"/>
      <arg name="usb_port_id"              value="$(arg usb_port_id)"/>
      <arg name="device_type"              value="$(arg device_type)"/>
      <arg name="json_file_path"           value="$(arg json_file_path)"/>

      <arg name="enable_pointcloud"        value="$(arg enable_pointcloud)"/>
      <arg name="pointcloud_texture_stream" value="$(arg pointcloud_texture_stream)"/>
      <arg name="pointcloud_texture_index"  value="$(arg pointcloud_texture_index)"/>
      <arg name="enable_sync"              value="$(arg enable_sync)"/>
      <arg name="align_depth"              value="$(arg align_depth)"/>

      <arg name="fisheye_width"            value="$(arg fisheye_width)"/>
      <arg name="fisheye_height"           value="$(arg fisheye_height)"/>
      <arg name="enable_fisheye"           value="$(arg enable_fisheye)"/>

      <arg name="depth_width"              value="$(arg depth_width)"/>
      <arg name="depth_height"             value="$(arg depth_height)"/>
      <arg name="enable_depth"             value="$(arg enable_depth)"/>

      <arg name="color_width"              value="$(arg color_width)"/>
      <arg name="color_height"             value="$(arg color_height)"/>
      <arg name="enable_color"             value="$(arg enable_color)"/>

      <arg name="infra_width"              value="$(arg infra_width)"/>
      <arg name="infra_height"             value="$(arg infra_height)"/>
      <arg name="enable_infra"            value="$(arg enable_infra)"/>
      <arg name="enable_infra1"            value="$(arg enable_infra1)"/>
      <arg name="enable_infra2"            value="$(arg enable_infra2)"/>

      <arg name="fisheye_fps"              value="$(arg fisheye_fps)"/>
      <arg name="depth_fps"                value="$(arg depth_fps)"/>
      <arg name="infra_fps"                value="$(arg infra_fps)"/>
      <arg name="color_fps"                value="$(arg color_fps)"/>
      <arg name="gyro_fps"                 value="$(arg gyro_fps)"/>
      <arg name="accel_fps"                value="$(arg accel_fps)"/>
      <arg name="enable_gyro"              value="$(arg enable_gyro)"/>
      <arg name="enable_accel"             value="$(arg enable_accel)"/>

      <arg name="publish_tf"               value="$(arg publish_tf)"/>
      <arg name="tf_publish_rate"          value="$(arg tf_publish_rate)"/>

      <arg name="filters"                  value="$(arg filters)"/>
      <arg name="clip_distance"            value="$(arg clip_distance)"/>
      <arg name="linear_accel_cov"         value="$(arg linear_accel_cov)"/>
      <arg name="initial_reset"            value="$(arg initial_reset)"/>
      <arg name="unite_imu_method"         value="$(arg unite_imu_method)"/>
      <arg name="topic_odom_in"            value="$(arg topic_odom_in)"/>
      <arg name="calib_odom_file"          value="$(arg calib_odom_file)"/>
      <arg name="publish_odom_tf"          value="$(arg publish_odom_tf)"/>
      <arg name="allow_no_texture_points"  value="$(arg allow_no_texture_points)"/>
    </include>
  </group>
</launch>

[skiddattle]

To get it to stop freezing, I set the gyro and accel to lower fps and it seems to stop the freezing issue: arg name="gyro_fps" default="200" arg name="accel_fps" default="63"

The curious thing is that I have the same setup in a different environment which is on the jetson xavier nx running ubuntu 18.04 but it does not have the freezing issue when I set it to the rates of 400hz and 250hz respectively. Is this an issue with the installation?

[skiddattle]

I would like to see if I can resolve this issue while maintaining the gyro and accel rates at the maximum possible of 400 and 250, as I would like to use it for visual inertial odometry and it is important for accuracy.

[MartyG-RealSense]

Thanks so much for your updates, including details about your method for stopping freezing.

Some explanation of the definition of frequencies for accel and gyro is provided in the link below:

https://github.com/IntelRealSense/realsense-ros/issues/729

[MartyG-RealSense]

Hi @skiddattle Do you require further assistance with this case, please? Thanks!

[skiddattle]

No need, thanks!

[MartyG-RealSense]

Thanks very much @skiddattle

Hi @zhizunhu As you created this case, do you still need help?

[ericpairet]

Hi @MartyG-RealSense,

I observe the same issues as reported by @zhizunhu on a Raspberry Pi 4. On top of that, sometimes the ROS wrapper stops publishing IMU data indefinitely despite the node remains alive (permanent blackout). As mentioned by @zhizunhu, there's no loss on the visual frames.

My setup: I initially intended to use Intel's official pre-built image for the Pi but I could not get the ROS wrapper working properly. Then, I tried two different setups of (librealsense - ROS wrapper): (2.38 - 2.2.17) and (2.39 - 2.2.18), following these instructions for the librealsense installation on the Pi. Both setups seem to run, but with IMU data loss and, eventually, permanent blackout.

I tried reducing the FPS of all devices to the minimum, just leaving on the accel and gyro (without fusing the data, and fusing their data with unite_imu_method=copy and unite_imu_method=linear_interpolation), and to force an initial_reset=true at startup. Any of these tests resulted in a consistent flow of IMU data, and unfortunately, I couldn't see any pattern.

Thanks in advance for any help!

[MartyG-RealSense]

Hi @ericpairet

@skiddattle above had the same problem, and did not require further assistance after being given a link that explained about IMU frequencies. Have you visited this link too please?

https://github.com/IntelRealSense/realsense-ros/issues/729

[ericpairet]

Hi @MartyG-RealSense,

Thanks for your fast response. I checked #729, but it doesn't clarify to me the temporal and permanent blackouts on the gyro, accel and imu topics. As both issues also happen with the raw data from the gyro and accel, I believe it might not be a problem related to unite_imu_method. I cannot replicate the issues on my desktop machine (the ROS wrapper works flawlessly in there), so if you think that the problem might be related with the Raspberry Pi setup, I'm happy to open a new ticket.

Below there's an example of the data flow I have with my current setup (gyro data, but I can confirm it happens with the accel too):

$ rostopic hz /camera/gyro/sample -w 200
subscribed to [/camera/accel/sample]
average rate: 402.496
min: 0.000s max: 0.025s std dev: 0.00277s window: 200
average rate: 397.038
min: 0.000s max: 0.024s std dev: 0.00291s window: 200
average rate: 403.295
min: 0.000s max: 0.031s std dev: 0.00371s window: 200
no new messages
average rate: 79.371
min: 0.000s max: 2.119s std dev: 0.14974s window: 200
average rate: 402.004
min: 0.000s max: 0.023s std dev: 0.00257s window: 200
average rate: 397.415
min: 0.000s max: 0.021s std dev: 0.00235s window: 200
average rate: 402.149
no new messages
no new messages
no new messages

Also, I observed that whenever there's a loss of data (or exactly when there's no new messages), the stamp on the messages rolls-back in time.

[doronhi]

Tracked in DSO-16098

[MartyG-RealSense]

@doronhi Thanks very much for referencing the DSO number.

@zhizunhu @ericpairet Basically, a DSO means that Intel have a report filed about an issue, so it should get looked at by Intel at some point. So it is probably best to wait for the outcome of the DSO, though I cannot offer a time estimate for that outcome.

[ericpairet]

Thanks @doronhi for chipping in and @MartyG-RealSense for the details.

Is the DSO-16098 strictly related to the usage of the camera with embedded systems such as the Raspberry Pi? Asking as I cannot replicate the issue described above in either a Desktop or laptop machine; I cannot see any message loss in either setup (checked up to the first 100k messages).

The inconsistency across platforms makes me think that this issue might result from bandwidth/processing limitations due to (a)~system without kernel patching (despite libUVC-backend) or (b)~system with limited computational power (the message loss is more significant as more sensors from the camera are enabled and the Pi gets busier).

Thinking that it might be some part of the installation that contributes to this issue, do we know of any image for the rasbperry pi, with librealsense and realsense-ros, that is not having this issue? I initially tried Intel's official pre-built image for the Pi but I could not get the ROS wrapper working properly.

[doronhi]

Thank you @ericpairet for the additional information and insight. I'll make a point to include the Intel's Raspberry Pi image into the considerations. I agree that frame drops are more common as resources drop low but why do time stamps go backwards require some inquiry.

[MartyG-RealSense]

Hi @ericpairet Do you require further assistance please? Thanks!

[ericpairet]

Hi @MartyG-RealSense, for what I understand, we cannot do much about frame drops and stamps going backwards. Just please update this issue if there is an outcome to the DSO-16098 or another solution to these problems is ever found.

[MartyG-RealSense]

Hi @ericpairet I will give this case an Enhancement tag to remind myself to keep this case open whilst DSO-16098 is active.

[etienn8]

Hi @MartyG-RealSense ! I encountered the same issue as @ericpairet. I'm also using a Raspberry Pi 4 and was wandering if you had any update about DSO-16098?

[MartyG-RealSense]

Hi @etienn8 I cannot comment on the current status of DSOs. The first public mention about the contents of specific DSOs is usually in the librealsense SDK's version release notes.

https://github.com/IntelRealSense/librealsense/wiki/Release-Notes

[KlemenDEV]

Are there any updates on this?

[MartyG-RealSense]

Hi @KlemenDEV There are currently no updates available about this issue.

[asimonov]

hello. I see the issue with same D455 on Intel laptop, but not on Jetson NX. would be great to have an answer eventually why this happens and how to guarantee it is not happening.