Generate camera-to-camera error metrics script

[aaguiar96]

Hi @miguelriemoliveira and @eupedrosa

This also has to done right? Should I reuse some code? Which script?

[aaguiar96]

EDIT: Actually, the train dataset has 8 complete collections. Do you think that it is enough?

[aaguiar96]

I'm obtaining this result:

Partial detection removed: label from collection 24, sensor left_camera
Partial detection removed: label from collection 25, sensor right_camera
Partial detection removed: label from collection 26, sensor left_camera
Partial detection removed: label from collection 27, sensor left_camera
Partial detection removed: label from collection 20, sensor left_camera
Partial detection removed: label from collection 21, sensor left_camera
Partial detection removed: label from collection 23, sensor left_camera
Partial detection removed: label from collection 28, sensor left_camera
Partial detection removed: label from collection 29, sensor left_camera
Partial detection removed: label from collection 40, sensor left_camera
Partial detection removed: label from collection 3, sensor left_camera
Partial detection removed: label from collection 2, sensor left_camera
Partial detection removed: label from collection 5, sensor left_camera
Partial detection removed: label from collection 4, sensor left_camera
Partial detection removed: label from collection 7, sensor left_camera
Partial detection removed: label from collection 6, sensor left_camera
Partial detection removed: label from collection 9, sensor left_camera
Partial detection removed: label from collection 8, sensor left_camera
Partial detection removed: label from collection 39, sensor left_camera
Partial detection removed: label from collection 38, sensor left_camera
Partial detection removed: label from collection 11, sensor left_camera
Partial detection removed: label from collection 10, sensor left_camera
Partial detection removed: label from collection 13, sensor left_camera
Partial detection removed: label from collection 12, sensor left_camera
Partial detection removed: label from collection 14, sensor left_camera
Partial detection removed: label from collection 19, sensor right_camera
Partial detection removed: label from collection 18, sensor left_camera
Partial detection removed: label from collection 31, sensor left_camera
Partial detection removed: label from collection 30, sensor left_camera
Partial detection removed: label from collection 37, sensor left_camera
Partial detection removed: label from collection 36, sensor left_camera
Partial detection removed: label from collection 35, sensor left_camera
Partial detection removed: label from collection 34, sensor right_camera
Partial detection removed: label from collection 33, sensor right_camera

Using 8 collections.

---------------------
 Starting stereo calibration ...

---------------------
 Done!

------
Calibration results:
------

('K_left', array([[698.66497803,   0.        , 653.06500244],
       [  0.        , 698.66497803, 368.27999878],
       [  0.        ,   0.        ,   1.        ]]))
('D_left', array([[-0.173342],
       [ 0.027032],
       [ 0.      ],
       [ 0.      ],
       [ 0.      ]]))
('K_right', array([[701.20001221,   0.        , 638.44500732],
       [  0.        , 701.20001221, 390.79598999],
       [  0.        ,   0.        ,   1.        ]]))
('D_right', array([[-0.176512],
       [ 0.029319],
       [ 0.      ],
       [ 0.      ],
       [ 0.      ]]))
('R', array([[ 9.99985899e-01,  5.14839359e-03, -1.30219567e-03],
       [-5.28011904e-03,  9.37703370e-01, -3.47396762e-01],
       [-5.67461995e-04,  3.47398739e-01,  9.37717332e-01]]))
('T', array([[-0.00030256],
       [-0.00021806],
       [-0.00013896]]))
('E', array([[-6.09994712e-07,  5.45505081e-05, -2.52756656e-04],
       [-1.39131991e-04,  1.04393170e-04,  2.83895739e-04],
       [ 2.19657646e-04, -2.82587883e-04,  1.04824043e-04]]))
('F', array([[-1.19467104e-08,  1.06836848e-06, -3.84420504e-03],
       [-2.72489183e-06,  2.04453406e-06,  4.91120455e-03],
       [ 4.08905397e-03, -5.36185857e-03,  1.00000000e+00]]))

It does no seem correct. The translation vector T does not have the 12cm baseline. Any ideas?...

[eupedrosa]

Yes, 8 collections is short. Do you want to try with my dataset? I will send you by email

[aaguiar96]

Yes @eupedrosa, it works well with yours! What do you think I should do? I can consider two datasets on the paper, and generate all the results for the two datasets. And, in OpenCV use only this one, since it is the only suitable... The opposite will happen with ICP, that will only work with my dataset, since it is the only one that have ZED's pointcloud.

Partial detection removed: label from collection 42, sensor left_camera
Partial detection removed: label from collection 48, sensor left_camera
Partial detection removed: label from collection 43, sensor left_camera
Partial detection removed: label from collection 49, sensor left_camera
Partial detection removed: label from collection 24, sensor left_camera
Partial detection removed: label from collection 26, sensor left_camera
Partial detection removed: label from collection 27, sensor left_camera
Partial detection removed: label from collection 20, sensor left_camera
Partial detection removed: label from collection 22, sensor right_camera
Partial detection removed: label from collection 23, sensor left_camera
Partial detection removed: label from collection 46, sensor left_camera
Partial detection removed: label from collection 47, sensor left_camera
Partial detection removed: label from collection 44, sensor left_camera
Partial detection removed: label from collection 45, sensor left_camera
Partial detection removed: label from collection 28, sensor left_camera
Partial detection removed: label from collection 0, sensor left_camera
Partial detection removed: label from collection 3, sensor left_camera
Partial detection removed: label from collection 2, sensor right_camera
Partial detection removed: label from collection 4, sensor left_camera
Partial detection removed: label from collection 51, sensor left_camera
Partial detection removed: label from collection 19, sensor left_camera
Partial detection removed: label from collection 18, sensor right_camera
Partial detection removed: label from collection 35, sensor left_camera
Partial detection removed: label from collection 50, sensor left_camera

Using 32 collections.

---------------------
 Starting stereo calibration ...

---------------------
 Done!

------
Calibration results:
------

('K_left', array([[695.16652355,   0.        , 654.54732098],
       [  0.        , 694.86289415, 370.53243436],
       [  0.        ,   0.        ,   1.        ]]))
('D_left', array([[-1.67330854e-01],
       [ 1.44714930e-02],
       [ 4.87055165e-04],
       [-9.55469245e-05],
       [ 8.53117222e-03]]))
('K_right', array([[701.55817538,   0.        , 639.44664198],
       [  0.        , 701.14474549, 394.25430151],
       [  0.        ,   0.        ,   1.        ]]))
('D_right', array([[-0.17194624],
       [ 0.01946158],
       [ 0.00033234],
       [ 0.00032733],
       [ 0.00666489]]))
('R', array([[ 9.99851036e-01,  2.65913812e-05,  1.72599477e-02],
       [-7.12695493e-05,  9.99996649e-01,  2.58793604e-03],
       [-1.72598210e-02, -2.58878064e-03,  9.99847687e-01]]))
('T', array([[-0.1214965],
       [-0.0002559],
       [ 0.0079809]]))
('E', array([[ 4.98558090e-06, -7.98020969e-03, -2.76514942e-04],
       [ 5.88270224e-03, -3.14315556e-04,  1.21615741e-01],
       [ 2.64520743e-04, -1.21496083e-01, -3.10008345e-04]]))
('F', array([[-2.14828578e-09,  3.44017327e-06, -1.19046037e-03],
       [-2.53634986e-06,  1.35577586e-07, -3.48411996e-02],
       [ 9.21375597e-04,  3.44912102e-02,  1.00000000e+00]]))

[eupedrosa]

Yes, you can consider multiple datasets. In fact, it is good for us. We less information we can achieve the same or better results.

[aaguiar96]

Ok, I'm working on this. For characterization (there are a lot of results and table lines) I will leave it as it is, only with my dataset. For comparison of atom with other methods, I'll add your dataset, and in this way our problems are solved.

Thanks! :)

[eupedrosa]

By the way, the benchmark ir running for the noise, 1000 runs for each level of percentage.

[aaguiar96]

That's really nice! Thanks!

[eupedrosa]

This may take a week or more in some cases. But I think we have time.

[aaguiar96]

I will write the other sections of results I leave that for the end.

The only missing results are ICP and those of the impact of the initial guess.

[aaguiar96]

Hi @miguelriemoliveira and @eupedrosa

FINALLY I was able to generate OpenCV results. The script is pretty generic! With an input json and two cameras, it calibrate them, and re-writes a json with the not anchored sensor transformation updated, using the sensor-to-sensor calibration and the function that I posted here before.

Here's the result with @eupedrosa's training dataset, and the generic test dataset that I'm using.

Starting evalutation...
If you enabled the visualization mode - press [SPACE] to advance between images
                                      - to stop visualization press [c] or [q]

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err
---------------------------------------------------------------------------
  0      -       0.9056    0.5055    0.4712    0.3146    2.1675    0.2585
  1      -       0.7119    0.7395    0.4589    0.4033    4.2591    0.2488
  2      -       0.6449    0.5882    0.4672    0.3297    4.8985    0.3097
  3      -       0.3431    0.4104    0.2807    0.2133    2.6878    0.2662
  4      -       0.6986    0.2599    0.3361    0.2374    3.7521    0.4207
  5      -       0.5183    0.4006    0.3559    0.3030    2.4333    0.0883
  6      -       0.3762    0.7973    0.4350    0.7256    3.9496    0.6507
  7      -       0.3509    0.2113    0.1826    0.2538    1.2665    0.0904
  8      -       0.3397    0.1959    0.2018    0.2448    0.7848    0.2340
  9      -       0.9157    2.6740    0.8845    1.1871    4.4536    0.6659
 10      -       0.5889    1.1050    0.4849    0.3743    5.0557    0.3441
 11      -       0.3927    0.4512    0.5738    0.4237    4.1770    0.1468
 12      -       0.6100    0.2040    0.4422    0.2630    1.3030    0.3052
 13      -       0.3620    0.2013    0.2698    0.2292    1.6152    0.2438
 14      -       0.8313    0.4824    0.3666    0.2091    2.2453    0.6286
---------------------------------------------------------------------------
 All   0.8628    0.5824    0.6216    0.6482    0.9658    0.0030    0.0057
---------------------------------------------------------------------------

Now I'll do the same for factory's calibration, and only ICP is missing!

[aaguiar96]

You can test it like this:

rosrun atom_evaluation cv_stereo_calib.py -json "/home/andreaguiar/Documents/datasets/deprecated/eurico_dataset/data_collected.json" -lc "left_camera" -rc "right_camera"

Since @eupedrosa's dataset is old, and do not have some features recently added, you have to comment these two lines: https://github.com/lardemua/atom/blob/27447a0589106e9aec8c15037459415909c5f549/atom_core/src/atom_core/dataset_io.py#L102-L105

[miguelriemoliveira]

Hi guys,

great work. Sorry for not answering before, this afternoon I was offline.

[aaguiar96]

Thanks @miguelriemoliveira

Closing this since it's solved.