ATOM calibration evaluation

[danifpdra]

Hi @aaguiar96,

Is this the right way to use this?

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/dataset_15_03_2021_2/atom_calibration.json -test_json $ATOM_DATASETS/dataset_11_03_2021/data_collected.json -ss world_camera -ts world_camera

The json train file is the atom_calibration and the test is the data_collected? What should be the source and target sensor? They should be the same? Can you please give me a more detailed explanation?

Thank you in advance.

[aaguiar96]

Hi @danifpdra

To generate the results, we use two different datasets:

• Train dataset: atom_calibration.json of the dataset used to obtain the calibration
• Test dataset: data_collection.json of a dataset taken for evaluation

So, you should record two different datasets, use one of them to calibrate the system, and then input both in the evaluation.

We do this so that we can test the calibration result in different pattern/sensor configurations (in the same way is done in machine/deep learning).

The source and target sensors are the sensors on which you will evaluate the calibration. For example, in the mmtbot for the camera-to-camera scenario you have the hand_camera and the world_camera right?

We evaluate the entire multi-sensor and multimodal calibration using pair of sensors. The idea is that you cover all the combinations with the atom_evaluation package.

[aaguiar96]

Thus, the proper way to run the camera-to-camera evaluation is something like this:

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/<train_dataset>/atom_calibration.json -test_json $ATOM_DATASETS/<test_dataset>/data_collected.json -ss hand_camera -ts world_camera

[danifpdra]

Hi @aaguiar96 ,

Thanks for the explanation. I have two datasets, yes.

So this is right? Are the results ok? I don't know what is expected.

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/dataset_15_03_2021_2/atom_calibration.json -test_json $ATOM_DATASETS/dataset_11_03_2021/data_collected.json -ss hand_camera -ts world_camera

Starting evalutation... If you enabled the visualization mode - press [SPACE] to advance between images

• to stop visualization press [c] or [q]

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
  0      -      84.3758    8.8509   12.1871    1.5613   158.6034  13.7441  
  1      -      91.2972    4.5127   10.3435    3.8889   183.6124  13.7376  
  2      -      18.2586   46.3385   11.5663    5.6244   161.2039  16.0081  
  3      -      16.6428   73.6595    9.9375    6.7719   208.5109  15.9847  
  4      -      16.6428   73.6595    9.9375    6.7719   208.5109  15.9847  
  5      -      18.7807   193.1756  18.2882   19.6044   342.0856  24.1302  
  6      -      17.0473   178.6826  17.7116   18.2010   342.0620  24.1288  
  7      -      31.3396   35.5751    3.1091    9.8344   141.9693  10.2006  
  8      -      44.8591   108.3143   9.5627   13.7066   271.5437  18.9171  
  9      -      45.0190   125.7274   9.3551   13.6472   271.3852  18.8846  
 10      -      66.1209   66.9513   12.8848   15.9621   195.7250  18.9477  
 11      -      84.8250   130.1541  35.8231   45.3340   402.5000  28.4434  
 12      -      39.1423   50.5343    6.1339    6.2484   102.3729   6.1246  
 13      -      31.0570   45.1196    1.9276    5.3206   102.3320   6.1719  
 14      -      15.4793   170.3707  18.6633   13.6836   277.8892  17.4750  
 15      -      13.5123   183.6354  14.3068   13.1444   338.2504  17.4432  
 16      -      16.7477   53.3467   19.1566   26.8295   183.4918  19.3131  
 17      -      29.1069   56.9960   30.8548   31.9760   184.1704  20.5203  
 18      -      27.5689   58.2606    5.3022    8.6206   142.2959  14.8078  
 19      -      18.5745   34.4202    6.1157    8.4515   135.5458  14.8426  
 20      -      52.1551    7.5411   14.2465    8.7825   174.5749  17.5327  
 21      -      26.4869   17.0725   16.9950   19.5482   139.0232  23.1161  
 22      -      196.7798  246.4605  14.7755   33.4563   543.6177  53.3385  
 23      -      86.4152   68.1038   52.5663   79.0303   298.0777  49.7470  
 24      -      75.1061   93.5803   50.2586   102.4539  263.9079  49.7076  
 25      -      52.3156   66.0627   51.7067   77.1201   359.2185  49.8546  
 26      -      53.8493   86.9729   58.6025   91.3184   519.6182  49.7316  
---------------------------------------------------------------------------
 All  88.2966   44.7219   86.0596   60.2671   94.1295    0.2464    0.4065  
---------------------------------------------------------------------------

[aaguiar96]

I @danifpdra

The results are really bad. All of them are in pixels, except the translation and rotation errors... So, the error is huge.

What calibration did you use? @miguelriemoliveira told me he obtained some good calibrations for mmtbot. Did you try with one of @miguelriemoliveira's calibrations?

[aaguiar96]

You should get errors not greater than 2/3 pixels...

This means that your calibration is not accurate.

[danifpdra]

The final calibration errors were really small, I don't know what happened.

[aaguiar96]

The final calibration errors were really small, I don't know what happened.

Ok. So, maybe the problem is in the test dataset.

Did you make sure that the arm was not moving when you recorded the collections?

Try to set the --test_json equal to the --train_json just to test the calibration in the collections used to calibrate.

[danifpdra]

Is the final line (All) the mean of the 29 lines?

[aaguiar96]

Look here:

https://github.com/lardemua/atom/blob/5541ab5d73e7f8463aaf462b28562d262fbb0e25/atom_evaluation/scripts/camera_to_camera_evalutation.py#L369-L374

The last line is the average x and y errors as well as their standard deviations. Also, the average translation and rotation errors.

[aaguiar96]

Just be careful with the last two columns.

If you see here, in the lines the translation and rotation errors are in mm and degrees. In the final line they are in meters and radians.

https://github.com/lardemua/atom/blob/5541ab5d73e7f8463aaf462b28562d262fbb0e25/atom_evaluation/scripts/camera_to_camera_evalutation.py#L319-L322

If you want, you can change all to meters and radians and commit. :-)

[danifpdra]

Thank you, I will look it up tomorrow and get back to you.

[danifpdra]

Hi @aaguiar96,

I tested this code in both datasets (the one made by me and the one made by @miguelriemoliveira) with the same dataset for training and testing and both results were bad. But both datasets give good final errors when running calibration.

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/dataset_11_03_2021/atom_calibration.json -test_json $ATOM_DATASETS/dataset_11_03_2021/data_collected.json -ss hand_camera -ts world_camera

   #    RMS      X err     Y err     X std     Y std     T err     R err   

  0      -       0.4100    0.3037    0.1385    0.0745    0.8737    0.1255  
  1      -       0.2678    0.2290    0.1802    0.0796    1.0792    0.1355  
  2      -      60.1002   48.9308    8.1247    3.3762   153.3574   2.3508  
  3      -      57.0308   51.8990    7.3912    3.3790   157.0408   2.3239  
  4      -      57.0308   51.8990    7.3912    3.3790   157.0408   2.3239  
  5      -      79.9414   180.1448  12.1695   13.3283   321.1123  10.8919  
  6      -      84.3008   177.3485  14.8761   13.1387   321.1117  10.8926  
  7      -      56.1021   51.1894    4.3115    4.6154   150.1099   4.3233  
  8      -      29.5748   92.5557    7.5856    5.3635   180.9925   5.4386  
  9      -      28.2918   96.6364    6.0142    5.8151   180.9986   5.3988  
 10      -      36.9518   91.6818    8.9445    7.1586   152.2810   5.4631  
 11      -      29.8699   99.1033   31.6475   46.8339   256.7668  22.5386  
 12      -      52.9188   52.2768   11.2002    4.5873   147.1061  12.2793  
 13      -      59.0464   58.4465   10.2534    7.6808   158.3568  12.3119  
 14      -      84.4638   180.7215   6.3145   13.0889   307.1741   3.9153  
 15      -      72.6385   165.7197   5.0641   12.9514   317.0609   3.9023  
 16      -      84.1845   88.1090   12.7585   24.5440   235.7862  23.3117  
 17      -      50.0790   29.7282   23.3812   31.0375   33.0558   20.3163  
 18      -      27.2271   87.1530    2.8691    3.5309   186.1407   7.8305  
 19      -      33.4116   90.8488    3.3927    4.7356   201.8145   7.8939  
 20      -      116.4101  22.8247    7.4983    6.1684   292.8299   6.4546  
 21      -      101.1814  38.6163   16.1984   25.0619   289.9202  15.0599  
 22      -      282.2283  276.8797  13.4869   14.8413   658.4631  41.3802  
 23      -      213.4203  95.5630   50.5710   96.0254   480.7017  45.1452  
 24      -      206.9499  96.1064   49.8517   109.9381  448.4914  45.1323  
 25      -      72.9487   74.8031   51.8638   79.4362   437.6794  45.2553  
 26      -      126.6258  95.2588   59.6580   94.7695   599.3865  45.1462  

 All  103.6291  73.6697   88.8666   65.0568   99.3684    0.2528    0.2634  

And mine: rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/dataset_15_03_2021_2/atom_calibration.json -test_json $ATOM_DATASETS/dataset_15_03_2021_2/data_collected.json -ss hand_camera -ts world_camera

  #     RMS      X err     Y err     X std     Y std     T err     R err   

  0      -       0.4100    0.3037    0.1385    0.0745    0.8737    0.1255  
  1      -       0.2678    0.2290    0.1802    0.0796    1.0792    0.1355  
  2      -      60.1002   48.9308    8.1247    3.3762   153.3574   2.3508  
  3      -      57.0308   51.8990    7.3912    3.3790   157.0408   2.3239  
  4      -      57.0308   51.8990    7.3912    3.3790   157.0408   2.3239  
  5      -      79.9414   180.1448  12.1695   13.3283   321.1123  10.8919  
  6      -      84.3008   177.3485  14.8761   13.1387   321.1117  10.8926  
  7      -      56.1021   51.1894    4.3115    4.6154   150.1099   4.3233  
  8      -      29.5748   92.5557    7.5856    5.3635   180.9925   5.4386  
  9      -      28.2918   96.6364    6.0142    5.8151   180.9986   5.3988  
 10      -      36.9518   91.6818    8.9445    7.1586   152.2810   5.4631  
 11      -      29.8699   99.1033   31.6475   46.8339   256.7668  22.5386  
 12      -      52.9188   52.2768   11.2002    4.5873   147.1061  12.2793  
 13      -      59.0464   58.4465   10.2534    7.6808   158.3568  12.3119  
 14      -      84.4638   180.7215   6.3145   13.0889   307.1741   3.9153  
 15      -      72.6385   165.7197   5.0641   12.9514   317.0609   3.9023  
 16      -      84.1845   88.1090   12.7585   24.5440   235.7862  23.3117  
 17      -      50.0790   29.7282   23.3812   31.0375   33.0558   20.3163  
 18      -      27.2271   87.1530    2.8691    3.5309   186.1407   7.8305  
 19      -      33.4116   90.8488    3.3927    4.7356   201.8145   7.8939  
 20      -      116.4101  22.8247    7.4983    6.1684   292.8299   6.4546  
 21      -      101.1814  38.6163   16.1984   25.0619   289.9202  15.0599  
 22      -      282.2283  276.8797  13.4869   14.8413   658.4631  41.3802  
 23      -      213.4203  95.5630   50.5710   96.0254   480.7017  45.1452  
 24      -      206.9499  96.1064   49.8517   109.9381  448.4914  45.1323  
 25      -      72.9487   74.8031   51.8638   79.4362   437.6794  45.2553  
 26      -      126.6258  95.2588   59.6580   94.7695   599.3865  45.1462  

 All  103.6291  73.6697   88.8666   65.0568   99.3684    0.2528    0.2634 

Can you help me here? Are both datasets bad?

[aaguiar96]

Hi @danifpdra

Can you post here both datasets, please? I will check the problem today, but only at night.

I will let you know when I have some news.

[danifpdra]

datasets.zip

Here it is. Sorry for the trouble.

[aaguiar96]

Hi @danifpdra

Sorry, I was not able to look at this today, and tomorrow I'll not have time to. Only thursday...

Meanwhile, @miguelriemoliveira if you have some time can you look at this?

This is what I would do:

• Test the evaluation script in a dataset collected in the agrob scenario (use one of the old datasets from our paper): it must work.
• Re-test the evaluation with mmtbot: first, test would be to use the same train and test dataset. If it does not work, something should be different in the new json files...

I can do this thursday. :)

[miguelriemoliveira]

Hi @aaguiar96 ,

sorry, I back to teaching so its been terrible. I will try to do what you ask (actually I wanted to learn this) but cannot promise to do it until tomorrow. If I can I will post.

[aaguiar96]

Hi @miguelriemoliveira and @danifpdra

If you want we can meet tomorrow for 1 hour to solve this. Are you available?

[miguelriemoliveira]

Sure. In the afternoon around 14h30?

• Miguel Riem de Oliveira * Diretor de curso Mestrado Eng. Automação Industrial email: @. | @. office: 22.2.18 | ext: 23887 Dep. de Engenharia Mecânica, Universidade de Aveiro Campus Universitário de Santiago, 3810-193 Portugal

On Wed, 17 Mar 2021 at 22:37, André Aguiar @.***> wrote:

Hi @miguelriemoliveira https://github.com/miguelriemoliveira and @danifpdra https://github.com/danifpdra

If you want we can meet tomorrow for 1 hour to solve this. Are you available?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lardemua/atom/issues/266#issuecomment-801486796, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVXUMPVLU76TYCSX4JDTEEVJBANCNFSM4ZHNAMXQ .

[danifpdra]

Yes, for me 14h30 is OK.

[aaguiar96]

Yes, deal.

[miguelriemoliveira]

Hi @aaguiar96 and @danifpdra ,

I just ran on my machine:

rosrun atom_evaluation camera_to_camera_evalutation.py -train_json $ATOM_DATASETS/mmtbot/dataset_11_03_2021/data_collected.json -test_json $ATOM_DATASETS/mmtbot/dataset_11_03_2021/data_collected.json -ss hand_camera -ts world_camera

Here are the results (@danifpdra this is a better way of showing the results, see how I did it)

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
  0      -       3.4404    1.9505    0.4705    0.5249   14.8582    0.0717  
  1      -       3.2244    2.5593    0.4492    0.6294   14.9576    0.0673  
  2      -      63.9032   46.5350    7.9987    3.0646   146.6212   2.4679  
  3      -      59.8366   49.4956    7.2074    3.0937   149.8668   2.4409  
  4      -      59.8366   49.4956    7.2074    3.0937   149.8668   2.4409  
  5      -      83.3211   178.4711  12.4155   13.8089   317.6061  11.0083  
  6      -      88.0355   175.4744  15.2502   13.5729   317.6072  11.0090  
  7      -      58.9794   54.2593    3.7488    4.3671   153.5318   4.2066  
  8      -      32.5737   90.3348    7.6458    5.5597   172.9828   5.5547  
  9      -      30.6492   94.4163    5.9011    5.9330   172.9975   5.5149  
 10      -      41.0670   88.7928    8.7942    7.2916   143.5875   5.5790  
 11      -      29.0455   97.3043   31.8490   46.8948   243.4104  22.5994  
 12      -      56.0775   55.1298   10.9723    4.0366   151.6922  12.1630  
 13      -      61.8171   61.4621    9.9352    7.5050   164.3805  12.1954  
 14      -      88.3163   178.9300   6.9079   13.5760   308.5460   4.0316  
 15      -      75.8038   164.1221   5.3635   13.4639   318.0763   4.0192  
 16      -      86.5208   90.8829   12.7332   24.5205   245.0205  23.2867  
 17      -      52.4828   28.9102   23.4537   31.2434   45.1116   20.3355  
 18      -      30.2174   89.4017    2.9213    3.1725   192.2879   7.7296  
 19      -      35.2838   92.7521    2.9770    4.3372   204.3418   7.7931  
 20      -      119.3615  24.8879    7.9495    6.6399   304.8158   6.4136  
 21      -      103.3976  40.6593   16.3926   25.0944   299.5797  15.0122  
 22      -      287.2159  274.3533  14.3145   14.9832   655.3267  41.3879  
 23      -      218.9312  94.4062   50.0496   96.1492   479.4726  45.0974  
 24      -      212.3624  95.7675   49.4666   110.3841  447.6000  45.0845  
 25      -      76.0896   73.6995   51.7108   79.4847   429.3704  45.2072  
 26      -      130.3397  93.9656   59.4381   94.8370   591.8426  45.0983  
---------------------------------------------------------------------------
 All  104.5022  76.7269   88.5614   65.5655   99.5856    0.2532    0.2636  
---------------------------------------------------------------------------

I am only looking at the error in pixels, which should be very close to zero (I used the same dataset) and in fact its around 70 pixels ... something is not well ...

[aaguiar96]

Hi @miguelriemoliveira and @danifpdra

b82abbd implemented the solution suggested by Miguel, but it does not seem to have impact.

I do not have any more time today. If you want to take a look... :) Otherwise, tomorrow I'll get back to it.

[miguelriemoliveira]

I am on it.

[miguelriemoliveira]

Got it. It was a matter of generalizing some more code in which you were using the selected_collection_key.

Results are actually very nice!!

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
  0      -       0.1540    0.0895    0.1674    0.0774    0.8228    0.1206  
  1      -       0.2230    0.0915    0.1578    0.0802    0.8455    0.1243  
  2      -       0.1210    0.0809    0.1488    0.0971    1.0258    0.1069  
  3      -       0.1315    0.0938    0.1512    0.1002    1.2477    0.1265  
  4      -       0.1315    0.0938    0.1512    0.1002    1.2477    0.1265  
  5      -       0.1050    0.1055    0.1080    0.0913    1.5560    0.0537  
  6      -       0.1119    0.0669    0.1138    0.0730    1.5510    0.0562  
  7      -       0.2267    0.0823    0.1079    0.0976    1.0169    0.0830  
  8      -       0.0977    0.0882    0.1010    0.0723    1.0959    0.0736  
  9      -       0.1713    0.1100    0.1234    0.0808    1.0892    0.1097  
 10      -       0.1971    0.1176    0.1208    0.1066    1.2078    0.0447  
 11      -       0.1480    0.3127    0.1541    0.1804    1.7340    0.1315  
 12      -       0.2654    0.1392    0.2173    0.1549    1.1609    0.1189  
 13      -       0.2616    0.1281    0.1758    0.1366    1.3278    0.1164  
 14      -       0.2033    0.1478    0.1772    0.1098    2.0135    0.0808  
 15      -       0.1615    0.0863    0.1630    0.1018    1.7786    0.1251  
 16      -       0.4162    0.0783    0.1698    0.0965    1.6477    0.1621  
 17      -       0.2987    0.0802    0.1652    0.0933    1.4910    0.1712  
 18      -       0.1024    0.1100    0.1026    0.1116    0.4532    0.0704  
 19      -       0.1175    0.1526    0.1142    0.1195    0.6548    0.0933  
 20      -       0.2418    0.1057    0.1162    0.1040    1.3093    0.0926  
 21      -       0.3358    0.0854    0.1239    0.0647    1.1025    0.1313  
 22      -       0.7411    0.2750    0.1925    0.6458    2.7424    0.0835  
 23      -       0.1844    0.3180    0.2157    0.1541    2.0009    0.1101  
 24      -       0.2416    0.4465    0.2661    0.6714    1.0779    0.2350  
 25      -       1.4700    0.5339    0.2522    0.6868    2.8441    0.1433  
 26      -       0.4329    0.2232    0.1810    0.1580    1.8090    0.1128  
---------------------------------------------------------------------------
 All   0.3192    0.2494    0.1459    0.3441    0.2593    0.0014    0.0019  
---------------------------------------------------------------------------

@aaguiar96 when you have time can you do the improvements in #268? Then I guess we can start transporting the results to the paper.

[miguelriemoliveira]

The avg of the Terr column does not make any sense ... different units?

[aaguiar96]

Great @miguelriemoliveira

I'll look at your commit and do the same for the LiDAR-camera case as soon as possible.

[aaguiar96]

The avg of the Terr column does not make any sense ... different units?

Yes. The All are in SI but the others not... Sorry :-)

[miguelriemoliveira]

Ahh, this is a deliberate, evil, approach to confuse people and make them feel bad and question their ability to average some values ... worked with me :)

[aaguiar96]

Hi @miguelriemoliveira

Is this correct?

https://github.com/lardemua/atom/blob/dbb24c47b9b8a3bbd61b3c870b7e4471b32ff2d3/atom_evaluation/scripts/camera_to_camera_evalutation.py#L178-L184

You want to copy to the test_dataset right?

[aaguiar96]

Ahh, this is a deliberate, evil, approach to confuse people and make them feel bad and question their ability to average some values ... worked with me :)

Muahahahahah!

[miguelriemoliveira]

I'll look at your commit and do the same for the LiDAR-camera case as soon as possible.

Thanks. #269

[miguelriemoliveira]

Is this correct?

No. Will commit a fix in a minute.

It worked nonetheless because my train dataset was optimized without optimizing the intrinsics, so they were the same in both datasets.

[aaguiar96]

No. Will commit a fix in a minute.

It worked nonetheless because my train dataset was optimized without optimizing the intrinsics, so they were the same in both datasets.

Right! :)

[miguelriemoliveira]

Is this complete? Please close if so ...

[miguelriemoliveira]

You are definitely not "closers" : - )