[imu&camera calibrate] runtime oveor::The time sequence must be nondecreasing. time 5 was not greater than or equal to time 4

[Bestboy125]

i just use the kalibr to caliibrate the imu and camera.

my imu is the ttl-485 wit HWT905-485,just one request and one answer.

first i just record a rosbag by subing the /imu and /usb_image_rect_raw.and one is 150hz another is 4hz.

second i can calibrate.

i think my rosbag time has no problem.but it show me that "The time sequence must be nondecreasing".I have no idea.

how can i check it?

my rosbag:https://drive.google.com/file/d/1zHDIAAe0t-u1Fy-oYFS9e21o_LimseFS/view?usp=sharing

while i run

rosrun kalibr kalibr_calibrate_imu_camera --target /home/april_s.yaml  --cam ./result/camera_calibration-camchain.yaml --imu ./result/imu.yaml --bag ./bag/camera_imu_calibration.bag  --bag-from-to 5 130

in shell the error will show

Initializing a pose spline with 14936 knots (100.000000 knots per second over 149.355158 seconds)
Traceback (most recent call last):
  File "/home/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in <module>
    main()
  File "/home/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 200, in main
    verbose = parsed.verbose)
  File "/home/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 105, in buildProblem
    cam.findTimeshiftCameraImuPrior(self.ImuList[0], verbose)
  File "/home/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 221, in findTimeshiftCameraImuPrior
    poseSpline = self.initPoseSplineFromCamera( timeOffsetPadding=0.0 )
  File "/home/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 315, in initPoseSplineFromCamera
    pose.initPoseSplineSparse(times, curve, knots, 1e-4)
RuntimeError: [Exception] /home/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/bsplines/src/BSpline.cpp:975: initSplineSparse() assert(times[i-1] <= times[i]) failed [1.68404e+09 <= 1.68404e+09]: The time sequence must be nondecreasing. time 5 was not greater than or equal to time 4

so i guess the problem is the time between the imu and camera. i check the rosbag info

path:        ./bag/camera_imu_calibration.bag
version:     2.0
duration:    2:42s (162s)
start:       May 14 2023 14:51:41.29 (1684043501.29)
end:         May 14 2023 14:54:24.06 (1684043664.06)
size:        518.5 MB
messages:    14378
compression: none [585/585 chunks]
types:       sensor_msgs/Image [060021388200f6f0f447d0fcd9c64743]
             sensor_msgs/Imu   [6a62c6daae103f4ff57a132d6f95cec2]
topics:      /imu                  13794 msgs    : sensor_msgs/Imu  
             /usb_image_rect_raw     584 msgs    : sensor_msgs/Image
---

and i check the /imu info by using ros_readbagfile this the part of the info

# timestamp (sec): 1684043501.291285276
# timestamp (sec): 1684043501.303655624
# timestamp (sec): 1684043501.310484409
# timestamp (sec): 1684043501.323313475
# timestamp (sec): 1684043501.334561825
# timestamp (sec): 1684043501.346612692
# timestamp (sec): 1684043501.358899117
# timestamp (sec): 1684043501.370918989
# timestamp (sec): 1684043501.382922411
# timestamp (sec): 1684043501.396509409
# timestamp (sec): 1684043501.408326149
# timestamp (sec): 1684043501.420297861
# timestamp (sec): 1684043501.427345276
# timestamp (sec): 1684043501.439752579
# timestamp (sec): 1684043501.451758862

and i check the camera rostopic by using ros_readbagfile this is a part of the info

# timestamp (sec): 1684043501.335471392
# timestamp (sec): 1684043501.610888243
# timestamp (sec): 1684043501.895060062
# timestamp (sec): 1684043502.170825481
# timestamp (sec): 1684043502.454141140
# timestamp (sec): 1684043502.734703779
# timestamp (sec): 1684043503.014587641
# timestamp (sec): 1684043503.294746637
# timestamp (sec): 1684043503.574270248
# timestamp (sec): 1684043503.849276543
# timestamp (sec): 1684043504.135670185
# timestamp (sec): 1684043504.412160158
# timestamp (sec): 1684043504.696390390
# timestamp (sec): 1684043504.971601725
# timestamp (sec): 1684043505.251634836
# timestamp (sec): 1684043505.531780720
# timestamp (sec): 1684043505.812109470
# timestamp (sec): 1684043506.092427492
# timestamp (sec): 1684043506.372832775
# timestamp (sec): 1684043506.653466225
# timestamp (sec): 1684043506.932847500
# timestamp (sec): 1684043507.216564417
# timestamp (sec): 1684043507.489185810
# timestamp (sec): 1684043507.773962498
# timestamp (sec): 1684043508.053948879
# timestamp (sec): 1684043508.334367990
# timestamp (sec): 1684043508.614395857
# timestamp (sec): 1684043508.894433498
# timestamp (sec): 1684043509.174345732
# timestamp (sec): 1684043509.455199003
# timestamp (sec): 1684043509.724263668

so i do not know whether it is due to time.if it is true,i guess i should use the message_fliter?just like this:http://wiki.ros.org/message_filters

Thank you!

[Bestboy125]

or just because of the frequence of camera

[goldbattle]

From the error, it looks like the messages are out of order. Maybe you should investigate this, and see if there is a driver problem / plot the timestamps to see where this problem is occuring.

[goldbattle]

Here is a starting point (not sure if that correct, but you can debug)

Script:

import rosbag
import matplotlib.pyplot as plt

bagfile = 'camera_imu_calibration.bag'
topic_time = '/usb_image_rect_raw'

# load bag message timestamps
timestamps = []
for topic, msg, t in rosbag.Bag(bagfile).read_messages():
    if topic != topic_time:
        continue
    timestamps.append(msg.header.stamp.to_sec())

# subtract start time to help viz a bit better
timestart = timestamps[0]
for i in range(0, len(timestamps)):
    timestamps[i] = timestamps[i] - timestart

# color invalid points if decreases
colors = ['black']
for i in range(1, len(timestamps)):
    if timestamps[i-1] < timestamps[i]:
        colors.append('black')
    else:
        colors.append('red')

# plot!
plt.scatter(range(0, len(timestamps)), timestamps, color=colors)
plt.plot(range(0, len(timestamps)), timestamps, '-k')
plt.xlabel('message number')
plt.ylabel('dataset time (s)')
plt.show()

[Bestboy125]

Thank you for your code and advice! According to your beautiful code,i check my rosbag.it looks like that the imu has no problem and the problem is my camera. i use the usb_cam to connect my camera.i guess i should turn up the frequence of camera to 20hz.or i should check the config of the usb_cam,such as io method.

[CQnancy]

感谢您的代码和建议！ 根据你漂亮的代码，我检查了我的rosbag。看起来imu没有问题，问题是我的相机。 我使用usb_cam连接我的相机。我想我应该将相机的频率调至20hz。或者我应该检查usb_cam的配置，例如io方法。

Hello,may I ask how you resolved the issue in the end?