Adding a new camera calibration tool for comparision

[manuelgitgomes]

@miguelriemoliveira suggested this repo, which is based on opencv: "Based on OpenCV, an open source calibration toolbox has been developed to support non-experts in calibrating single and stereo camera systems and to improve the calibration results." - from their article I will search for more options

[miguelriemoliveira]

Hi @manuelgitgomes ,

I was searching around quite a lot and could not find anything.

The only other is kalibr:

https://github.com/ethz-asl/kalibr/wiki/installation

but we have used in the past with poor results

https://www.researchgate.net/publication/341152457_A_ROS_Framework_for_the_Extrinsic_Calibration_of_Intelligent_Vehicles_a_Multi-Sensor_Multi-Modal_Approach

I will call you.

[miguelriemoliveira]

Hi @eupedrosa ,

we searched and could not find anything other than opencv. So if you can get results with kalibr that would be nice.

[eupedrosa]

Hi @miguelriemoliveira.

I guess we can use kalibr, it now compiles with noetic. To calibrate I need a ROS bag. The bag can be created this way: https://github.com/ethz-asl/kalibr/wiki/bag-format

Can you provide the images and the pattern dimensions?

[miguelriemoliveira]

Hi @eupedrosa ,

Thanks for the help.

From what I see above all you need are the datasets, since from then you can generate the csvs and later the kalibr rosbags, right?

In the paper the table we want to change is Table 3: Performance comparison of methods for RGB to RGB camera evaluation.

which means we only have to provide kalibr results in the datasets where it was possible to do so with opencv, i.e. atlascar-1,

atlascar-1 Not sure in the table why we do not have the resutls for atlascar-1 and atlascar-2 as in the other cases.

I cannot find the datasets, I think @danifpdra should have them. Do you?

agrob-2 $ATOM_DATASETS/agrob/agrob_paper/test_dataset

My suggestion @eupedrosa is that you get started with the agrob dataset and then we will get you the atlscar2 datasets(s).

[danifpdra]

Hi,

Yes, I do have the atlascar datasets. Here they are atlascar2_t.zip

[eupedrosa]

@danifpdra , the _test dataset is the one used for evaluating, correct?

[danifpdra]

@danifpdra , the _test dataset is the one used for evaluating, correct?

yes, correct

[miguelriemoliveira]

ai ai ai, these names. We have a dataset renaming tool working. After this paper is submitted we should change the name.

[danifpdra]

ai ai ai, these names. We have a dataset renaming tool working. After this paper is submitted we should change the name.

why do you say that? it's well named

[eupedrosa]

Are you sure that the _test dataset is the one used for the evaluation in the paper? The results that I got using this one is very different from the ones in the paper.

For example, the center-camera to left-camera has a RMSE of 3 pixels in the paper, using the _test for evaluating Kalibr I got an RMSE of ~1.0. The defiference is very high.

[danifpdra]

Are you sure that the _test dataset is the one used for the evaluation in the paper? The results that I got using this one is very different from the ones in the paper.

For example, the center-camera to left-camera has a RMSE of 3 pixels in the paper, using the _test for evaluating Kalibr I got an RMSE of ~1.0. The defiference is very high.

I am absolutely sure... I saved all the commands are results that I took at that time.

The used command was (for that pair, for example):

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/atlascar2_train_1/atom_calibration.json -test_json $ATOM_DATASETS/atlascar2_test_1/data_collected.json -ts top_left_camera -ss top_center_rgbd_camera_rgb

And I got

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
Removing collection 0 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 1 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 2 -> pattern was not found in sensor top_left_camera (must be found in all sensors).
Removing collection 3 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 4 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 5 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 6 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 7 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 8 -> pattern was not found in sensor top_left_camera (must be found in all sensors).
Removing collection 9 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 11 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 12 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 13 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 14 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 15 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 16 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 19 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 20 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
 10      -       4.2218    2.9960    2.1744    1.1420   48.8765    3.6995  
 17      -       2.4942    2.2247    1.4607    0.8224   38.1231    4.6451  
 18      -       3.2638    1.7202    1.8568    1.1989   47.4546    7.1151  
---------------------------------------------------------------------------
 All   3.2830    3.3266    2.3136    3.8323    1.2133    0.0448    0.0899  
---------------------------------------------------------------------------

[miguelriemoliveira]

@eupedrosa , perhaps use the old camera_to_camera evaluaton just to make sure its the same script.

https://github.com/lardemua/atom/blob/noetic-devel/atom_evaluation/scripts/old/camera_to_camera_evaluation_old.py

perhaps that accounts for some difference.

[miguelriemoliveira]

why do you say that? it's well named

yes it is. I was @eupedrosa that wrote just _test and I thought that was one of the datasets... sorry.

[eupedrosa]

@danifpdra, do you also have the command for the right camera to center camera?

[danifpdra]

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/atlascar2_train_1/atom_calibration.json -test_json $ATOM_DATASETS/atlascar2_test_1/data_collected.json -ss top_right_camera -ts top_center_rgbd_camera_rgb

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
Removing collection 0 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 1 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 2 -> pattern was not found in sensor top_right_camera (must be found in all sensors).
Removing collection 3 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 4 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 5 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 6 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 7 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 8 -> pattern was not found in sensor top_right_camera (must be found in all sensors).
Removing collection 9 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 11 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 12 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 13 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 14 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 15 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 16 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 19 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
Removing collection 20 -> pattern was not found in sensor top_center_rgbd_camera_rgb (must be found in all sensors).
 10      -       3.1614    2.6731    0.9793    0.3433   58.3537    3.0959  
 17      -       0.9429    2.5899    1.0688    0.6577   89.5472    6.0827  
 18      -       0.9033    2.3093    1.0956    0.9849   73.2524    7.3129  
---------------------------------------------------------------------------
 All   2.3751    1.6692    2.5241    1.7741    0.7334    0.0737    0.0959  
---------------------------------------------------------------------------

[eupedrosa]

Thanks @danifpdra.

@miguelriemoliveira, I finished calibrating with Kalibr. I was not able to use extract the correct data from the full calibration with Kalibr. I had to do pairwise calibration.

Here is the data:

[danifpdra]

@eupedrosa ,

Can you tell me how did you obtain these results? Did you create a script in ATOM to replicate this for other calibration systems or did you just replicate kalibr? I wanted to replicate it for Larcc

[eupedrosa]

I had two create 2 scripts:

• dataset_to_bag.py: will create a ROS bag from an ATOM dataset.
• kalibr_stereo_calib.py: will read the ATOM dataset and the Kalibr output file and will create kalibr_calibration.json that can be used with the ATOM toolset.

Right now it assumes that we are calibrating stereo. This may be changed to accept any number of cameras.

I will commit the scripts.

[danifpdra]

Thank you very much! I will try it out.

[miguelriemoliveira]

Thanks @eupedrosa ,

great work. I think this is a big step in getting the paper accepted.

I will try to complete the rest this week.

[danifpdra]

@eupedrosa ,

Can you tell me which parameters you used to run kalibr?

[eupedrosa]

Sorry for the late reply @danifpdra.

Here is an example for atlascar

rosrun kalibr kalibr_calibrate_cameras --models pinhole-radtan pinhole-radtan pinhole-radtan --topics /top_left_camera/image_raw /top_right_camera/image_raw --target target.yaml --bag output.bag

And the target.yaml is

target_type: 'checkerboard' #gridtype
targetCols: 9               #number of internal chessboard corners
targetRows: 6              #number of internal chessboard corners
rowSpacingMeters: 0.101      #size of one chessboard square [m]
colSpacingMeters: 0.101      #size of one chessboard square [m]