search

ethz-asl / kalibr

The Kalibr visual-inertial calibration toolbox
Other
4.43k stars 1.4k forks source link

Calibrating camera imu fails - There must be at least one time segment #602

Closed AreteQin closed 1 year ago

AreteQin commented 1 year ago

Hello there,

Thank you for sharing this excellent work!

I went through other similar issues as mine but I still met this problem #407. I got the same errors and I checked my bag file using rqt_bag as shown below: image

Then I tried to fix it using restamp_bag, and I got this in rqt_bag: image

Finally, I used the output bag file for calibrating, but I still receive the same error:

rosrun kalibr kalibr_calibrate_imu_camera --target '/home/qin/Downloads/CAM_IMU_Calibration/april_6x6_80x80cm.yaml' --imu '/home/qin/Downloads/CAM_IMU_Calibration/imu.yaml' --cam '/home/qin/Downloads/CAM_IMU_Calibration/D435_BGR8_640_480_30_intrinsics.yaml' --bag '/home/qin/Downloads/CAM_IMU_Calibration/outbag.bag'  
importing libraries
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Initializing IMUs:
  Update rate: 2500.0
  Accelerometer:
    Noise density: 0.0009221746732053619 
    Noise density (discrete): 0.046108733660268096 
    Random walk: 4.389173826627885e-06
  Gyroscope:
    Noise density: 0.0009221746796214183
    Noise density (discrete): 0.04610873398107091 
    Random walk: 4.3891738326059456e-06
Initializing imu rosbag dataset reader:
    Dataset:          /home/qin/Downloads/CAM_IMU_Calibration/outbag.bag
    Topic:            /qcar_imu/raw
    Number of messages: 366754
Reading IMU data (/qcar_imu/raw)
  Read 366754 imu readings over 139.8 seconds                         
Initializing calibration target:
  Type: aprilgrid
  Tags: 
    Rows: 6
    Cols: 6
    Size: 0.02 [m]
    Spacing 0.006 [m]
Initializing camera chain:
Camera chain - cam0:
  Camera model: pinhole
  Focal length: [607.592, 608.191]
  Principal point: [322.553, 242.127]
  Distortion model: radtan
  Distortion coefficients: [0, 0, 0, 0]
  baseline: no data available
Initializing camera rosbag dataset reader:
    Dataset:          /home/qin/Downloads/CAM_IMU_Calibration/outbag.bag
    Topic:            /D435/color
    Number of images: 4191
Extracting calibration target corners
  Extracted corners for 1355 images (of 4191 images)                              

Building the problem
    Spline order: 6
    Pose knots per second: 100
    Do pose motion regularization: False
        xddot translation variance: 1000000.000000
        xddot rotation variance: 100000.000000
    Bias knots per second: 50
    Do bias motion regularization: True
    Blake-Zisserman on reprojection errors -1
    Acceleration Huber width (sigma): -1.000000
    Gyroscope Huber width (sigma): -1.000000
    Do time calibration: True
    Max iterations: 30
    Time offset padding: 0.030000
Estimating time shift camera to imu:

Initializing a pose spline with 0 knots (100.000000 knots per second over 0.000000 seconds)
Traceback (most recent call last):
  File "/home/qin/kalibr_workspace/devel/lib/kalibr/kalibr_calibrate_imu_camera", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in <module>
    main()
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 188, in main
    iCal.buildProblem(splineOrder=6, 
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 105, in buildProblem
    cam.findTimeshiftCameraImuPrior(self.ImuList[0], verbose)
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 221, in findTimeshiftCameraImuPrior
    poseSpline = self.initPoseSplineFromCamera( timeOffsetPadding=0.0 )
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 315, in initPoseSplineFromCamera
    pose.initPoseSplineSparse(times, curve, knots, 1e-4)
RuntimeError: [Exception] /home/qin/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/bsplines/src/BSpline.cpp:970: initSplineSparse() assert(numSegments >= 1) failed [0 >= 1]: There must be at least one time segment

Any help would be greatly appreciated! Thank you

goldbattle commented 1 year ago

You can see from your second picture that your timestamps for the images are incorrect.

On Mon, Mar 27, 2023 at 11:56 AM AreteQin @.***> wrote:

Hello there,

Thank you for sharing this excellent work!

I went through other similar issues as mine but I still met this problem

407 https://github.com/ethz-asl/kalibr/issues/407. I got the same

errors and I checked my bag file using rqt_bag as shown below: [image: image] https://user-images.githubusercontent.com/38554510/227994822-712b5d04-699e-4b75-a4a8-5a6f032cfa3e.png

Then I tried to fix it using restamp_bag https://github.com/ethz-asl/dataset_tools/blob/master/python/restamp_bag.py, and I got this in rqt_bag: [image: image] https://user-images.githubusercontent.com/38554510/227995765-f4f4587a-517b-42b7-914d-39a00199b9e2.png

Finally, I used the output bag file for calibrating, but I still receive the same error:

rosrun kalibr kalibr_calibrate_imu_camera --target '/home/qin/Downloads/CAM_IMU_Calibration/april_6x6_80x80cm.yaml' --imu '/home/qin/Downloads/CAM_IMU_Calibration/imu.yaml' --cam '/home/qin/Downloads/CAM_IMU_Calibration/D435_BGR8_640_480_30_intrinsics.yaml' --bag '/home/qin/Downloads/CAM_IMU_Calibration/outbag.bag' importing libraries the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update' Initializing IMUs: Update rate: 2500.0 Accelerometer: Noise density: 0.0009221746732053619 Noise density (discrete): 0.046108733660268096 Random walk: 4.389173826627885e-06 Gyroscope: Noise density: 0.0009221746796214183 Noise density (discrete): 0.04610873398107091 Random walk: 4.3891738326059456e-06 Initializing imu rosbag dataset reader: Dataset: /home/qin/Downloads/CAM_IMU_Calibration/outbag.bag Topic: /qcar_imu/raw Number of messages: 366754 Reading IMU data (/qcar_imu/raw) Read 366754 imu readings over 139.8 seconds Initializing calibration target: Type: aprilgrid Tags: Rows: 6 Cols: 6 Size: 0.02 [m] Spacing 0.006 [m] Initializing camera chain: Camera chain - cam0: Camera model: pinhole Focal length: [607.592, 608.191] Principal point: [322.553, 242.127] Distortion model: radtan Distortion coefficients: [0, 0, 0, 0] baseline: no data available Initializing camera rosbag dataset reader: Dataset: /home/qin/Downloads/CAM_IMU_Calibration/outbag.bag Topic: /D435/color Number of images: 4191 Extracting calibration target corners Extracted corners for 1355 images (of 4191 images)

Building the problem Spline order: 6 Pose knots per second: 100 Do pose motion regularization: False xddot translation variance: 1000000.000000 xddot rotation variance: 100000.000000 Bias knots per second: 50 Do bias motion regularization: True Blake-Zisserman on reprojection errors -1 Acceleration Huber width (sigma): -1.000000 Gyroscope Huber width (sigma): -1.000000 Do time calibration: True Max iterations: 30 Time offset padding: 0.030000 Estimating time shift camera to imu:

Initializing a pose spline with 0 knots (100.000000 knots per second over 0.000000 seconds) Traceback (most recent call last): File "/home/qin/kalibr_workspace/devel/lib/kalibr/kalibr_calibrate_imu_camera", line 15, in exec(compile(fh.read(), python_script, 'exec'), context) File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in main() File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 188, in main iCal.buildProblem(splineOrder=6, File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 105, in buildProblem cam.findTimeshiftCameraImuPrior(self.ImuList[0], verbose) File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 221, in findTimeshiftCameraImuPrior poseSpline = self.initPoseSplineFromCamera( timeOffsetPadding=0.0 ) File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 315, in initPoseSplineFromCamera pose.initPoseSplineSparse(times, curve, knots, 1e-4) RuntimeError: [Exception] /home/qin/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/bsplines/src/BSpline.cpp:970: initSplineSparse() assert(numSegments >= 1) failed [0 >= 1]: There must be at least one time segment

Any help would be greatly appreciated! Thank you

— Reply to this email directly, view it on GitHub https://github.com/ethz-asl/kalibr/issues/602, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ6TYTDHIX5LNN24EXGVL3W6G2DTANCNFSM6AAAAAAWJKVODU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

AreteQin commented 1 year ago

@goldbattle I see that the second one is different from the first picture. Is it because the second one seems to have no messages in the image topic? But if I check the bag file using 'rosbag info', it still shows that there are two topics as shown below:

rosbag info '/home/qin/Downloads/CAM_IMU_Calibration/outbag.bag' 
path:        /home/qin/Downloads/CAM_IMU_Calibration/outbag.bag
version:     2.0
duration:    466632hr 14:53s (1679876093s)
start:       Dec 31 1969 19:00:00.00 (0.00)
end:         Mar 26 2023 20:14:53.15 (1679876093.15)
size:        3.7 GB
messages:    370945
compression: none [4192/4192 chunks]
types:       sensor_msgs/Image [060021388200f6f0f447d0fcd9c64743]
             sensor_msgs/Imu   [6a62c6daae103f4ff57a132d6f95cec2]
topics:      /D435/color       4191 msgs    : sensor_msgs/Image
             /qcar_imu/raw   366754 msgs    : sensor_msgs/Imu

Two bag files just give me the same error.

goldbattle commented 1 year ago

You should go read up on how message store time and how rosbags work. You can also see from what you posted that the rosbag shows that you have invalid timestamps with the start time being in 1969 (epoch).

duration:    466632hr 14:53s (1679876093s)
start:       Dec 31 1969 19:00:00.00 (0.00)

If you are using rqtbag, then you should be able to inspect the header, or echo just /D435/color/header to console with rostopic echo command.

AreteQin commented 1 year ago

@goldbattle Thank you so much for the tip.

But my original bag file seems to be fine, right?

path:        /home/qin/Downloads/CAM_IMU_Calibration/d435_IMU_CAM.bag
version:     2.0
duration:    2:19s (139s)
start:       Mar 26 2023 20:12:33.15 (1679875953.15)
end:         Mar 26 2023 20:14:53.15 (1679876093.15)
size:        3.7 GB
messages:    370945
compression: none [4192/4192 chunks]
types:       sensor_msgs/Image [060021388200f6f0f447d0fcd9c64743]
             sensor_msgs/Imu   [6a62c6daae103f4ff57a132d6f95cec2]
topics:      /D435/color       4191 msgs    : sensor_msgs/Image
             /qcar_imu/raw   366754 msgs    : sensor_msgs/Imu

I got the same error as following:

rosrun kalibr kalibr_calibrate_imu_camera --target '/home/qin/Downloads/CAM_IMU_Calibration/april_6x6_80x80cm.yaml' --imu '/home/qin/Downloads/CAM_IMU_Calibration/imu.yaml' --cam '/home/qin/Downloads/CAM_IMU_Calibration/D435_BGR8_640_480_30_intrinsics.yaml' --bag '/home/qin/Downloads/CAM_IMU_Calibration/d435_IMU_CAM.bag' 
importing libraries
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Initializing IMUs:
  Update rate: 2500.0
  Accelerometer:
    Noise density: 0.0009221746732053619 
    Noise density (discrete): 0.046108733660268096 
    Random walk: 4.389173826627885e-06
  Gyroscope:
    Noise density: 0.0009221746796214183
    Noise density (discrete): 0.04610873398107091 
    Random walk: 4.3891738326059456e-06
Initializing imu rosbag dataset reader:
    Dataset:          /home/qin/Downloads/CAM_IMU_Calibration/d435_IMU_CAM.bag
    Topic:            /qcar_imu/raw
    Number of messages: 366754
Reading IMU data (/qcar_imu/raw)
  Read 366754 imu readings over 139.8 seconds                         
Initializing calibration target:
  Type: aprilgrid
  Tags: 
    Rows: 6
    Cols: 6
    Size: 0.02 [m]
    Spacing 0.006 [m]
Initializing camera chain:
Camera chain - cam0:
  Camera model: pinhole
  Focal length: [607.592, 608.191]
  Principal point: [322.553, 242.127]
  Distortion model: radtan
  Distortion coefficients: [0, 0, 0, 0]
  baseline: no data available
Initializing camera rosbag dataset reader:
    Dataset:          /home/qin/Downloads/CAM_IMU_Calibration/d435_IMU_CAM.bag
    Topic:            /D435/color
    Number of images: 4191
Extracting calibration target corners
  Extracted corners for 1355 images (of 4191 images)                              

Building the problem
    Spline order: 6
    Pose knots per second: 100
    Do pose motion regularization: False
        xddot translation variance: 1000000.000000
        xddot rotation variance: 100000.000000
    Bias knots per second: 50
    Do bias motion regularization: True
    Blake-Zisserman on reprojection errors -1
    Acceleration Huber width (sigma): -1.000000
    Gyroscope Huber width (sigma): -1.000000
    Do time calibration: True
    Max iterations: 30
    Time offset padding: 0.030000
Estimating time shift camera to imu:

Initializing a pose spline with 0 knots (100.000000 knots per second over 0.000000 seconds)
Traceback (most recent call last):
  File "/home/qin/kalibr_workspace/devel/lib/kalibr/kalibr_calibrate_imu_camera", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in <module>
    main()
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 188, in main
    iCal.buildProblem(splineOrder=6, 
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 105, in buildProblem
    cam.findTimeshiftCameraImuPrior(self.ImuList[0], verbose)
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 221, in findTimeshiftCameraImuPrior
    poseSpline = self.initPoseSplineFromCamera( timeOffsetPadding=0.0 )
  File "/home/qin/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccSensors.py", line 315, in initPoseSplineFromCamera
    pose.initPoseSplineSparse(times, curve, knots, 1e-4)
RuntimeError: [Exception] /home/qin/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/bsplines/src/BSpline.cpp:970: initSplineSparse() assert(numSegments >= 1) failed [0 >= 1]: There must be at least one time segment

Any help would be greatly appreciated!

goldbattle commented 1 year ago

There are two times you need to know of. The message appended to the bag time and the message header time. The message header time is the sensor time that the sensor driver should be stamping and populating. The rosbag has a secondary time which just records when the messaged was written to disk. After you "cook" the bag, it will replace the rosbag timestamps with the ones that the sensor publishes which is what caused your second bag to look different in rqt_bag. Both bags are incorrect and cannot be used in Kalibr. Please look into the sensor driver you are using to ensure it correctly publishes ros timestamps in the header.

For example rqt_bag states:

The messages are shown at the timestamp stored in the bag file. This timestamp may differ from the message's Header timestamp (if any), e.g. rosbag record stores the time the message was received. https://wiki.ros.org/rqt_bag#Timeline

As before, please do some reading up on rosbag times. Here are some starting points. This isn't related to Kalibr. https://answers.ros.org/question/318536/understanding-rosbag-timestamps/?answer=318556#post-id-318556 https://wiki.ros.org/rosbag/Cookbook#Rewrite_bag_with_header_timestamps https://wiki.ros.org/Bags/Format/2.0#Message_data

Hope this helps.

AreteQin commented 1 year ago

@goldbattle Thank you so much for your explanation! After modifying the timestamp in the image publisher, I successfully fixed the problem and ran the calibration.