Best Practice for Calibration

[vhanded]

My calibration result is inconsistant, and not acceptable.

I am using 4 x Realsense D410 sensors, arrange in 4 corners, facing center. Sensors are 1.2 meters away side by side. All HW sync.

        1.2m
O <-----------> O
|               |
|   1.2m        |   1.2m
|               |
O <-----------> O
     1.2m

I run this command to start vicalib: ./vicalib -cam realsense2:[size=848x480,rgb=0,ir0=1,depth=0,emitter=0,exposure=33000,fps=90]// -grid_preset medium

I am using Infrared as sensor, as D410 has no RGB.

Use case: I use vicalib to extract all the camera poses, and transform each point clouds from realsense sensor to the camera poses I collected earlier.

Here is part of the calibration log:

I0613 23:25:46.874855  5885 vicalibrator.h:968] Reprojection error for camera 0: 437.973 rmse: 1.22636
I0613 23:25:46.916352  5885 vicalibrator.h:968] Reprojection error for camera 1: 458.594 rmse: 1.52307
I0613 23:25:46.960085  5885 vicalibrator.h:968] Reprojection error for camera 2: 517.38 rmse: 1.5021
I0613 23:25:47.000623  5885 vicalibrator.h:968] Reprojection error for camera 3: 388.42 rmse: 1.18741

From my understanding, to get a good calibration, we would need to keep Reprojection error low. The default for vicalib is 0.15, which means the error is less than a pixel?

In my case, I had NEVER achieve that standard after nearly 50 trials, the best is 0.28, but with false positive result.

Method I tried:

1. Install lights above the 4 sensors to increase brightness; Now all dots on grid are visible. Grid are sticked on a flat whiteboard, very sturdy.
2. Move gridboard slowly. (This would increase the chance of MSE not able to converge, and ended immaturely. But still random)
3. Move gridboard fast, MSE would converge nicely, but high reprojection error. (Sometimes not converge nicely)
4. Increase Frame Per Second , FPS
5. Set outlier_threshold to 1, 0.5, 0.25, with remove_outliers true. This would return low reprojection error, but wrong result.

What other method I can try to improve calibration?

Thanks.

[JStech]

Those reprojection errors are incredibly bad. Usually, a bad calibration still has single-digit reprojection error. Are you able to position the grid to occupy more than half of the frame?

Also, it's impossible to have an exposure of 33,000 (microseconds) and framerate of 90 Hz, as 90 Hz has only 11,000 microseconds per frame.

[vhanded]

@JStech Thanks for your reply.

After I posted that question above, I found out about the fps issue too. While the Realsense SDK wasn't complaining anything to me, I wasn't notice about it. (Sorry for lack of fundamental knowledge on this). So a 33000 exposure would have about 15 fps, according to realsense viewer.

Grid usually occupy the scene like this.

While turning the grid to another direction, the grid in 1 of the sensor turns red.

Grid only visible in a scene, while transform to another scene.

[vhanded]

I had updated the command:

./vicalib -cam realsense2:[size=1280x720,rgb=0,ir0=1,depth=0,emitter=0,exposure=30000,fps=30]// -grid_preset medium

Do sequence of sensors important? For example, I must show the grid to sensor A -> sensor B -> sensor C -> sensor D? Or random is fine?

While optimizing, there are about 50% chance, the poses keep diverging, and not converge, and ended prematurely, because of hitting function_convergence of 1^-6. Like the image in bottom left above.

Several guess: Could it be my grid too big?

[vhanded]

Several of typical reprojection error:

I0614 09:35:34.979439 17328 vicalibrator.h:968] Reprojection error for camera 0: 287.483 rmse: 0.806644
I0614 09:35:35.030995 17328 vicalibrator.h:968] Reprojection error for camera 1: 644.93 rmse: 2.1455
I0614 09:35:35.114843 17328 vicalibrator.h:968] Reprojection error for camera 2: 595.996 rmse: 1.45063
I0614 09:35:35.174999 17328 vicalibrator.h:968] Reprojection error for camera 3: 609.428 rmse: 1.71985

I0614 09:27:37.178468 16632 vicalibrator.h:968] Reprojection error for camera 0: 4896.19 rmse: 19.2952
I0614 09:27:37.203020 16632 vicalibrator.h:968] Reprojection error for camera 1: 3194.48 rmse: 14.4446
I0614 09:27:37.215054 16632 vicalibrator.h:968] Reprojection error for camera 2: 3414.89 rmse: 21.5825
I0614 09:27:37.227375 16632 vicalibrator.h:968] Reprojection error for camera 3: 1141.01 rmse: 7.386

I0614 09:16:26.780233 16299 vicalibrator.h:968] Reprojection error for camera 0: 1044.94 rmse: 1.54019
I0614 09:16:27.018033 16299 vicalibrator.h:968] Reprojection error for camera 1: 1019.07 rmse: 1.29818
I0614 09:16:27.058802 16299 vicalibrator.h:968] Reprojection error for camera 2: 33.8165 rmse: 0.451449
I0614 09:16:27.483989 16299 vicalibrator.h:968] Reprojection error for camera 3: 1459.98 rmse: inf

I0614 09:07:41.519497 15794 vicalibrator.h:968] Reprojection error for camera 0: 3101.6 rmse: 8.04599
I0614 09:07:41.571852 15794 vicalibrator.h:968] Reprojection error for camera 1: 18705.5 rmse: 57.6443
I0614 09:07:41.634307 15794 vicalibrator.h:968] Reprojection error for camera 2: 132770 rmse: 356.789
I0614 09:07:41.687201 15794 vicalibrator.h:968] Reprojection error for camera 3: 1641.29 rmse: 4.9132

I0614 08:53:41.142446 15168 vicalibrator.h:968] Reprojection error for camera 0: 4241.04 rmse: 14.0876
I0614 08:53:41.192937 15168 vicalibrator.h:968] Reprojection error for camera 1: 5796.79 rmse: 19.5876
I0614 08:53:41.245985 15168 vicalibrator.h:968] Reprojection error for camera 2: 6527.9 rmse: 19.3561
I0614 08:53:41.312386 15168 vicalibrator.h:968] Reprojection error for camera 3: 1989.87 rmse: 5.32008

Just for the sake of comparison, this is calibration result of only 1 sensor:

Solver Summary (v 2.0.0-eigen-(3.2.92)-lapack-suitesparse-(4.4.6)-cxsparse-(3.1.4)-eigensparse-no_openmp)

                                     Original                  Reduced
Parameter blocks                         1092                     1090
Parameters                               7637                     7630
Effective parameters                     6547                     6541
Residual blocks                        356233                   356233
Residuals                              712466                   712466

Minimizer                        TRUST_REGION

Sparse linear algebra library    SUITE_SPARSE
Trust region strategy                  DOGLEG (TRADITIONAL)

                                        Given                     Used
Linear solver          SPARSE_NORMAL_CHOLESKY   SPARSE_NORMAL_CHOLESKY
Threads                                     4                        4
Linear solver ordering              AUTOMATIC                     1090

Cost:
Initial                          2.426464e+06
Final                            9.706355e+04
Change                           2.329400e+06

Minimizer iterations                       22
Successful steps                           17
Unsuccessful steps                          5

Time (in seconds):
Preprocessor                         0.223964

  Residual only evaluation           1.122639 (22)
  Jacobian & residual evaluation    21.724264 (17)
  Linear solver                      2.923683 (17)
Minimizer                           27.862608

Postprocessor                        0.016556
Total                               28.103129

Termination:                      CONVERGENCE (Function tolerance reached. |cost_change|/cost: 8.009684e-07 <= 1.000000e-06)

I0614 09:51:59.044592 18105 vicalib-engine.cc:403] Finished... 
W0614 09:51:59.044620 18105 vicalib-task.cc:836] Reprojection error of 1.7818 was greater than maximum of 0.15 for camera 0
I0614 09:51:59.044770 18105 vicalibrator.h:537] ------------------------------------------
I0614 09:51:59.044778 18105 vicalibrator.h:539] Camera: 0
I0614 09:51:59.044807 18105 vicalibrator.h:540] 00953.256 00956.943 00630.128 00363.537 0.0517639 -0.206776 000.23823
I0614 09:51:59.044859 18105 vicalibrator.h:541] [ 1, 0, 0, 0; 0, 1, 0, 0; 0, 0, 1, 0; 0, 0, 0, 1 ]
I0614 09:51:59.044872 18105 vicalibrator.h:542] 

[JStech]

I've never tried a large multi-camera calibration like this, and I wonder if the problem is that the target is not continuously visible to the cameras. Would you be able to position and move the target so it is continuously visible to two cameras at a time? Then at least you could get the extrinsic calibrations in pairs. Or maybe hold the target flat so all four cameras can see it at once?

Just to be sure everything else is running right, you might consider doing a single-camera calibration (so it'd just be intrinsics), and see if you get a decent reprojection error.

[vhanded]

I've never tried a large multi-camera calibration like this, and I wonder if the problem is that the target is not continuously visible to the cameras. Would you be able to position and move the target so it is continuously visible to two cameras at a time? Then at least you could get the extrinsic calibrations in pairs. Or maybe hold the target flat so all four cameras can see it at once?

Just to be sure everything else is running right, you might consider doing a single-camera calibration (so it'd just be intrinsics), and see if you get a decent reprojection error.

I replied earlier with the summary of a single camera calibration. Anything wrong that you can see?

[JStech]

Sorry, missed that. It looks alright, although not sub-pixel. The 410 has a rolling shutter, right? More light, shorter exposure, and slower movements are always helpful. Although, getting the intrinsic calibration down under 1 won't solve your multi-camera issues.

[vhanded]

D410 is indeed a rolling shutter. Has anyone tried if this really impact badly?

If calibrate together with IMU device, will that become better? Besides the suggestion of short exposure and slow movements, any added technique to improve the accuracy?

Can a smaller grid board reduce the impact of rolling shutter?

Thanks.

[JStech]

Kalibr handles rolling shutter cameras, you might try it. I don't have personal experience with calibrating a rolling shutter camera, because our lab avoids using them.

I don't think an IMU would help. It doesn't contribute any information to the intrinsic calibration, and you'd have to move the cameras and leave the target stationary, which looks difficult for your setup. I don't think a smaller grid would help either.

To reiterate what I said, though, the rolling shutter might explain why your intrinsic calibration had a reprojection error of 1.78, but doesn't explain the terrible errors you're getting when you do the multi-camera extrinsic calibration. So fixing (or mitigating) the issues of a rolling shutter won't solve your original issue. I still recommend trying to calibrate just two adjacent cameras, using a sequence where they can both see the target the entire time.

[vhanded]

@JStech The IMU setup that I saw is from Intel, like this: https://youtu.be/VSHDyUXSNqY?t=890

Anyway, Kalibr is very interesting. I will give it a try.

[vhanded]

@JStech After tried out Kalibr, the rolling shutter calibration is for per camera. I would like to try the method of reduce the exposure with vicalib first.

This is the command I tried: vicalib -cam realsense2:[size=1280x720,rgb=0,ir0=1,depth=0,emitter=0,exposure=2000,gain=248,fps=6]// -grid_preset medium

And the output I got:

F0628 09:35:23.713708  9039 vicalib-engine.cc:128] Could not create camera from URI: realsense2:[size=1280x720,rgb=0,ir0=1,depth=0,emitter=0,exposure=2000,gain=248,fps=6]//. Reason: hwmon command 0x69 failed. Error type:  (1).
*** Check failure stack trace: ***
    @     0x7fc9250655cd  google::LogMessage::Fail()
    @     0x7fc925067433  google::LogMessage::SendToLog()
    @     0x7fc92506515b  google::LogMessage::Flush()
    @     0x7fc925067e1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x4a5491  visual_inertial_calibration::VicalibEngine::VicalibEngine()
    @           0x4a26d4  main
    @     0x7fc920dc4830  __libc_start_main
    @           0x4a3449  _start
    @              (nil)  (unknown)
Aborted (core dumped)

I found out, that the lowest exposure vicalib can achieve is 7551, with fps of 6. In realsense-viewer, I can go down to exposure 1, and a reasonable value that still can see the grid clearly, is 2000, with maximum gain of 248.

How can I achieve exposure of 2000 in vicalib?

Thanks.

[JStech]

I'm not sure that the exposure units are the same in realsense-viewer and the HAL URI. The exposure value in HAL is microseconds, so values are typically about 10000-30000.

[vhanded]

I'm not sure that the exposure units are the same in realsense-viewer and the HAL URI. The exposure value in HAL is microseconds, so values are typically about 10000-30000.

I found out the issue. Both HAL and realsense-viewer using microseconds, so it is not the cause.

Because I HW sync all the cameras, I modified the HAL code to auto assign master and slave to the cameras. However, the timing for setting master and slave should have at least 1000 milisecond apart, then it will be fine.