Analtino's Thesis

[miguelriemoliveira]

This is what is missing:

• [x] Record two bags files using the softbot simulation
• [x] Record two datasets, one train (dataset 3) and one test (dataset 4)
• [x] Correct auto annotations manually using dataset playback (in both datasets)
• [x] Calibrate train dataset and get atom_dataset.json and optimized xacro file
• [x] Produce manual rgb annotations for evaluation
• [x] Evaluate rgb camera to rgb camera
• [x] Evaluate left rgb camera to 3d lidar
• [x] Evaluate right rgb camera to 3d lidar

[miguelriemoliveira]

rgb to rgb

---------------------------------------------------------------------------
  #     RMS      X err     Y err     X std     Y std     T err     R err   
---------------------------------------------------------------------------
  0      -       0.1294    0.1135    0.1622    0.1323    0.8718    0.0774  
  1      -       0.1817    0.1219    0.1194    0.1441    1.5334    0.0524  
  2      -       0.3104    0.0996    0.1463    0.1235    1.6032    0.0353  
  3      -       0.1510    0.1904    0.1881    0.2312    2.3608    0.0746  
  4      -       0.3028    0.1693    0.1901    0.2009    2.6113    0.1287  
  5      -       0.4569    0.1544    0.1904    0.1548    1.5990    0.0557  
  6      -       0.6636    0.0991    0.1714    0.1204    1.4419    0.0349  
  7      -       0.2896    0.1080    0.1472    0.1335    0.5421    0.0671  
  8      -       0.2332    0.1221    0.1648    0.1279    0.6289    0.1023  
  9      -       0.1170    0.1239    0.1393    0.1508    0.2719    0.0407  
 10      -       0.1855    0.1300    0.1821    0.1635    1.0486    0.0776  
 11      -       0.1559    0.1212    0.1709    0.1697    0.5205    0.0275  
 12      -       0.1324    0.1615    0.1531    0.1979    0.2111    0.0159  
 13      -       0.1501    0.1343    0.1609    0.1871    0.2951    0.0151  
 14      -       0.2013    0.1172    0.0894    0.1529    0.4136    0.0364  
 15      -       0.1441    0.1260    0.1272    0.1577    0.6176    0.0392  
 16      -       0.1754    0.1100    0.1315    0.1440    0.4454    0.0221  
 17      -       0.0990    0.0989    0.1166    0.1232    0.6039    0.0505  
 18      -       0.2118    0.1356    0.1411    0.1396    1.2158    0.0782  
 19      -       0.1114    0.1229    0.1408    0.1380    0.1858    0.0400  
 20      -       0.0992    0.1117    0.1361    0.1303    0.6483    0.0337  
 21      -       0.1691    0.1022    0.1668    0.1257    2.0340    0.0857  
 22      -       0.1276    0.1051    0.1427    0.1208    1.3273    0.0610  
 23      -       0.3657    0.1482    0.1515    0.1343    0.9209    0.0313  
 24      -       0.1202    0.1019    0.1533    0.1235    1.9866    0.0789  
 25      -       0.1500    0.1282    0.1296    0.1502    1.4613    0.0251  
 26      -       0.1431    0.1024    0.1291    0.1256    1.2851    0.0353  
 27      -       0.1462    0.1039    0.1141    0.1379    2.0885    0.0563  
 28      -       0.1891    0.0978    0.1459    0.1251    0.8080    0.0334  
 29      -       0.1539    0.1117    0.1332    0.1460    0.8120    0.0683  
 30      -       0.1695    0.1158    0.1483    0.1488    2.5062    0.0757  
 31      -       0.1425    0.1450    0.1426    0.1724    1.1389    0.0479  
 32      -       0.1339    0.1416    0.1312    0.1681    1.1529    0.0701  
 33      -       0.1116    0.1464    0.1395    0.1728    1.3336    0.1271  
 34      -       0.1202    0.1522    0.1548    0.1596    3.9640    0.1868  
 35      -       0.1252    0.1647    0.1704    0.1757    3.6764    0.1740  
 36      -       0.1364    0.1698    0.1607    0.1522    2.2220    0.1114  
 37      -       0.1692    0.1337    0.1646    0.1551    2.6554    0.1000  
 38      -       0.1338    0.1280    0.1352    0.1458    2.3499    0.1367  
 39      -       0.1499    0.1368    0.1560    0.1646    1.0277    0.0403  
 40      -       0.1266    0.1569    0.1544    0.1599    1.4622    0.1059  
 41      -       0.1160    0.1435    0.1473    0.1369    0.6194    0.0828  
 42      -       0.1566    0.1250    0.1373    0.1340    1.0555    0.0275  
---------------------------------------------------------------------------
 All   0.2045    0.1810    0.1270    0.2354    0.1562    0.0013    0.0012  
---------------------------------------------------------------------------

[miguelriemoliveira]

Annotations with collection 7,21 or 22, 30 has problems

Test dataset (dataset 3) may have problems in the labels of 18, 23 and 24

[miguelriemoliveira]

evaluation lidar3d to front left camera

------------------------------------------------------------------------------------------------------------------------------------------------------------
            #                       RMS                    X Error                  Y Error           X Standard Deviation     Y Standard Deviation   
------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/mike/datasets/softbot/dataset3/front_left_camera_0.jpg
            0                        -                     2.7955                   2.2584                   3.5239                   2.5743          
/home/mike/datasets/softbot/dataset3/front_left_camera_1.jpg
            1                        -                     3.9782                   2.6021                   5.4622                   2.7563          
/home/mike/datasets/softbot/dataset3/front_left_camera_2.jpg
            2                        -                     3.9021                   2.7895                   4.6399                   3.1173          
/home/mike/datasets/softbot/dataset3/front_left_camera_3.jpg
            3                        -                     1.9562                   1.9177                   2.1408                   1.7624          
/home/mike/datasets/softbot/dataset3/front_left_camera_4.jpg
            4                        -                     2.6878                   2.4213                   3.1260                   1.9274          
/home/mike/datasets/softbot/dataset3/front_left_camera_5.jpg
            5                        -                     3.8574                   2.7606                   4.2845                   1.8888          
/home/mike/datasets/softbot/dataset3/front_left_camera_6.jpg
            6                        -                     4.5141                   3.5746                   4.3706                   2.3196          
/home/mike/datasets/softbot/dataset3/front_left_camera_7.jpg
            7                        -                     3.4871                   2.8491                   4.2329                   3.1062          
/home/mike/datasets/softbot/dataset3/front_left_camera_8.jpg
            8                        -                     3.3352                   2.4038                   4.0207                   2.2415          
/home/mike/datasets/softbot/dataset3/front_left_camera_9.jpg
            9                        -                     2.8363                   2.7760                   3.6288                   4.5567          
/home/mike/datasets/softbot/dataset3/front_left_camera_10.jpg
           10                        -                     4.1547                   3.1915                   6.1998                   5.5565          
/home/mike/datasets/softbot/dataset3/front_left_camera_11.jpg
           11                        -                     2.8024                   2.0703                   3.4717                   2.1637          
/home/mike/datasets/softbot/dataset3/front_left_camera_12.jpg
           12                        -                     3.5427                   2.6821                   6.1542                   4.3754          
/home/mike/datasets/softbot/dataset3/front_left_camera_13.jpg
           13                        -                     3.3913                   2.6443                   5.6921                   4.1771          
/home/mike/datasets/softbot/dataset3/front_left_camera_14.jpg
           14                        -                     6.1990                   4.9469                   9.9196                   7.0177          
/home/mike/datasets/softbot/dataset3/front_left_camera_15.jpg
           15                        -                     4.9163                   3.8412                   6.6870                   4.7728          
/home/mike/datasets/softbot/dataset3/front_left_camera_16.jpg
           16                        -                     3.0534                   2.6204                   3.7516                   1.5618          
/home/mike/datasets/softbot/dataset3/front_left_camera_17.jpg
           17                        -                     6.4503                   5.3659                   9.7940                   7.6708          
/home/mike/datasets/softbot/dataset3/front_left_camera_18.jpg
           18                        -                     2.1304                   1.7666                   2.8415                   1.3823          
/home/mike/datasets/softbot/dataset3/front_left_camera_19.jpg
           19                        -                     2.4099                   2.4339                   2.8632                   2.3593          
/home/mike/datasets/softbot/dataset3/front_left_camera_20.jpg
           20                        -                     4.4003                   3.5177                   7.4810                   5.6910          
/home/mike/datasets/softbot/dataset3/front_left_camera_21.jpg
           21                        -                     5.2308                   4.1138                   8.5293                   6.3724          
/home/mike/datasets/softbot/dataset3/front_left_camera_22.jpg
           22                        -                     6.3067                   4.5602                   10.1271                  6.2445          
/home/mike/datasets/softbot/dataset3/front_left_camera_23.jpg
           23                        -                     2.4232                   2.0202                   3.0268                   1.6952          
/home/mike/datasets/softbot/dataset3/front_left_camera_24.jpg
           24                        -                     29.9324                  23.1333                  21.8483                  22.9008         
/home/mike/datasets/softbot/dataset3/front_left_camera_25.jpg
           25                        -                     3.7693                   3.4371                   4.7935                   3.6731          
/home/mike/datasets/softbot/dataset3/front_left_camera_26.jpg
           26                        -                     1.9159                   1.8124                   2.4119                   2.0086          
/home/mike/datasets/softbot/dataset3/front_left_camera_27.jpg
           27                        -                     2.2179                   2.6051                   2.4963                   2.9742          
/home/mike/datasets/softbot/dataset3/front_left_camera_28.jpg
           28                        -                     2.0002                   2.0558                   2.1992                   1.5916          
/home/mike/datasets/softbot/dataset3/front_left_camera_29.jpg
           29                        -                     3.8291                   3.4642                   4.9847                   3.5781          
/home/mike/datasets/softbot/dataset3/front_left_camera_30.jpg
           30                        -                     3.1032                   2.6167                   3.8846                   2.5129          
/home/mike/datasets/softbot/dataset3/front_left_camera_31.jpg
           31                        -                     1.8994                   1.9275                   2.4141                   2.0367          
/home/mike/datasets/softbot/dataset3/front_left_camera_32.jpg
           32                        -                     2.5406                   2.2731                   3.2266                   2.2622          
/home/mike/datasets/softbot/dataset3/front_left_camera_33.jpg
           33                        -                     1.6625                   1.4650                   2.4240                   1.5091          
/home/mike/datasets/softbot/dataset3/front_left_camera_34.jpg
           34                        -                     2.4674                   2.1915                   3.2200                   2.0444          
/home/mike/datasets/softbot/dataset3/front_left_camera_35.jpg
           35                        -                     1.9513                   1.6645                   2.5201                   1.6465          
/home/mike/datasets/softbot/dataset3/front_left_camera_36.jpg
           36                        -                     2.0540                   1.7242                   2.9953                   2.3435          
/home/mike/datasets/softbot/dataset3/front_left_camera_37.jpg
           37                        -                     2.0096                   2.1018                   2.3434                   1.8021          
/home/mike/datasets/softbot/dataset3/front_left_camera_38.jpg
           38                        -                     3.3213                   3.2655                   4.1489                   3.8456          
/home/mike/datasets/softbot/dataset3/front_left_camera_39.jpg
           39                        -                     2.0653                   2.0981                   2.7365                   2.7009          
/home/mike/datasets/softbot/dataset3/front_left_camera_40.jpg
           40                        -                     1.9574                   1.4540                   2.4983                   1.5094          
/home/mike/datasets/softbot/dataset3/front_left_camera_41.jpg
           41                        -                     3.2505                   2.5428                   5.4081                   3.1445          
/home/mike/datasets/softbot/dataset3/front_left_camera_42.jpg
           42                        -                     3.1206                   2.6769                   4.6778                   2.4892          
------------------------------------------------------------------------------------------------------------------------------------------------------------
           All                    6.3214                   3.8722                   3.1780                   7.0725                   5.1511          
------------------------------------------------------------------------------------------------------------------------------------------------------------

[miguelriemoliveira]

3dlidar to front right camera

------------------------------------------------------------------------------------------------------------------------------------------------------------
            #                       RMS                    X Error                  Y Error           X Standard Deviation     Y Standard Deviation   
------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/mike/datasets/softbot/dataset3/front_right_camera_0.jpg
            0                        -                     3.2364                   2.4399                   3.9445                   2.0692          
/home/mike/datasets/softbot/dataset3/front_right_camera_1.jpg
            1                        -                     3.5492                   2.0239                   4.7209                   1.9173          
/home/mike/datasets/softbot/dataset3/front_right_camera_2.jpg
            2                        -                     2.6242                   2.2022                   2.9361                   1.8684          
/home/mike/datasets/softbot/dataset3/front_right_camera_3.jpg
            3                        -                     4.1346                   2.7751                   6.4065                   3.2583          
/home/mike/datasets/softbot/dataset3/front_right_camera_4.jpg
            4                        -                     3.0407                   2.8007                   3.6647                   2.5396          
/home/mike/datasets/softbot/dataset3/front_right_camera_5.jpg
            5                        -                     34.6752                  8.6937                   66.0614                  16.5196         
/home/mike/datasets/softbot/dataset3/front_right_camera_6.jpg
            6                        -                     4.3202                   2.9969                   4.8037                   2.1679          
/home/mike/datasets/softbot/dataset3/front_right_camera_7.jpg
            7                        -                     2.3092                   3.6279                   1.7432                   3.2220          
/home/mike/datasets/softbot/dataset3/front_right_camera_8.jpg
            8                        -                     4.6322                   3.4227                   7.7653                   3.3725          
/home/mike/datasets/softbot/dataset3/front_right_camera_9.jpg
            9                        -                     2.3869                   1.8232                   2.9751                   1.9661          
/home/mike/datasets/softbot/dataset3/front_right_camera_10.jpg
           10                        -                     1.9746                   2.9132                   1.7781                   2.4693          
/home/mike/datasets/softbot/dataset3/front_right_camera_11.jpg
           11                        -                     2.6407                   2.1116                   2.9377                   1.7538          
/home/mike/datasets/softbot/dataset3/front_right_camera_12.jpg
           12                        -                     2.5183                   1.6561                   3.1134                   1.7849          
/home/mike/datasets/softbot/dataset3/front_right_camera_13.jpg
           13                        -                     4.1112                   1.9809                   8.2358                   3.9146          
/home/mike/datasets/softbot/dataset3/front_right_camera_14.jpg
           14                        -                     2.3092                   1.9951                   3.1011                   2.3558          
/home/mike/datasets/softbot/dataset3/front_right_camera_15.jpg
           15                        -                     3.5375                   2.8322                   4.7112                   3.7561          
/home/mike/datasets/softbot/dataset3/front_right_camera_16.jpg
           16                        -                     3.0437                   2.0728                   4.5299                   2.5023          
/home/mike/datasets/softbot/dataset3/front_right_camera_17.jpg
           17                        -                     5.3259                   3.2675                   9.6710                   4.7539          
/home/mike/datasets/softbot/dataset3/front_right_camera_18.jpg
           18                        -                     2.6511                   4.7432                   3.3250                   9.2568          
/home/mike/datasets/softbot/dataset3/front_right_camera_19.jpg
           19                        -                     3.2599                   2.2105                   4.2781                   1.9616          
/home/mike/datasets/softbot/dataset3/front_right_camera_20.jpg
           20                        -                     3.4994                   2.3371                   5.1933                   2.6145          
/home/mike/datasets/softbot/dataset3/front_right_camera_21.jpg
           21                        -                     2.8447                   1.9559                   4.5495                   1.8912          
/home/mike/datasets/softbot/dataset3/front_right_camera_22.jpg
           22                        -                     3.3125                   2.0040                   4.3973                   1.4842          
/home/mike/datasets/softbot/dataset3/front_right_camera_23.jpg
           23                        -                     1.4485                   2.5560                   1.4090                   2.3799          
/home/mike/datasets/softbot/dataset3/front_right_camera_24.jpg
           24                        -                     33.8117                  19.8823                  22.3015                  20.7401         
/home/mike/datasets/softbot/dataset3/front_right_camera_25.jpg
           25                        -                     4.0737                   2.5149                   5.5955                   2.7176          
/home/mike/datasets/softbot/dataset3/front_right_camera_26.jpg
           26                        -                     3.5059                   1.7962                   6.0352                   1.9123          
/home/mike/datasets/softbot/dataset3/front_right_camera_27.jpg
           27                        -                     2.6342                   1.5541                   3.7161                   1.6294          
/home/mike/datasets/softbot/dataset3/front_right_camera_28.jpg
           28                        -                     2.9208                   1.9588                   3.9626                   2.0357          
/home/mike/datasets/softbot/dataset3/front_right_camera_29.jpg
           29                        -                     4.4600                   2.3166                   6.3168                   2.5392          
/home/mike/datasets/softbot/dataset3/front_right_camera_30.jpg
           30                        -                     4.6033                   3.1567                   7.3134                   4.2898          
/home/mike/datasets/softbot/dataset3/front_right_camera_31.jpg
           31                        -                     4.0144                   2.3677                   7.9736                   3.2703          
/home/mike/datasets/softbot/dataset3/front_right_camera_32.jpg
           32                        -                     3.5320                   2.0373                   4.8041                   2.2488          
/home/mike/datasets/softbot/dataset3/front_right_camera_33.jpg
           33                        -                     3.0367                   1.5916                   5.5477                   2.1395          
/home/mike/datasets/softbot/dataset3/front_right_camera_34.jpg
           34                        -                     4.3262                   2.3203                   7.0526                   2.8093          
/home/mike/datasets/softbot/dataset3/front_right_camera_35.jpg
           35                        -                     2.3205                   1.3923                   4.1004                   2.1847          
/home/mike/datasets/softbot/dataset3/front_right_camera_36.jpg
           36                        -                     3.1925                   1.6972                   5.1724                   1.9237          
/home/mike/datasets/softbot/dataset3/front_right_camera_37.jpg
           37                        -                     3.7147                   2.5906                   6.2955                   3.0633          
/home/mike/datasets/softbot/dataset3/front_right_camera_38.jpg
           38                        -                     7.7273                   4.1108                   17.8120                  6.7534          
/home/mike/datasets/softbot/dataset3/front_right_camera_39.jpg
           39                        -                     4.7115                   4.3699                   7.3145                   7.3209          
/home/mike/datasets/softbot/dataset3/front_right_camera_40.jpg
           40                        -                     2.1682                   1.3393                   3.1082                   1.7496          
/home/mike/datasets/softbot/dataset3/front_right_camera_41.jpg
           41                        -                     14.8101                  2.3354                   36.6347                  2.9055          
/home/mike/datasets/softbot/dataset3/front_right_camera_42.jpg
           42                        -                     5.7837                   2.7056                   11.2853                  3.6000          
------------------------------------------------------------------------------------------------------------------------------------------------------------
           All                    10.8237                  5.0894                   3.0023                   14.2626                  5.3824          
------------------------------------------------------------------------------------------------------------------------------------------------------------

[miguelriemoliveira]

Here is the calibrated dataset 4 and annotated dataset 3

https://drive.google.com/file/d/19Msv0142d8ZZsf68t4x4qsgHHU8xdTO3/view?usp=sharing

[manuelgitgomes]

These results seem strange...

I ran the rgb2rgb evaluation with the calibrated dataset 4 as train and the annotated dataset 3 as test and obtained the following results:

+--------------+-----------+-------------+-------------+------------+-----------+
| Collection # | RMS (pix) | X err (pix) | Y err (pix) | Trans (mm) | Rot (deg) |
+--------------+-----------+-------------+-------------+------------+-----------+
|      0       |   0.1925  |    0.1190   |    0.1075   |   0.4796   |   0.0704  |
|      1       |   0.3305  |    0.2762   |    0.1092   |   1.3536   |   0.0512  |
|      2       |   0.4572  |    0.4233   |    0.0890   |   1.4605   |   0.0475  |
|      3       |   0.3178  |    0.1528   |    0.1444   |   1.5434   |   0.0545  |
|      4       |   0.4001  |    0.3349   |    0.1331   |   0.8410   |   0.0779  |
|      5       |   0.5122  |    0.4783   |    0.0958   |   0.7874   |   0.0252  |
|      6       |   0.7867  |    0.7727   |    0.0700   |   1.2579   |   0.0233  |
|      7       |   0.3371  |    0.3051   |    0.0814   |   0.5426   |   0.0324  |
|      8       |   0.2772  |    0.2281   |    0.0879   |   0.3132   |   0.0587  |
|      9       |   0.3117  |    0.1443   |    0.0916   |   0.7438   |   0.0385  |
|      10      |   0.2455  |    0.1723   |    0.1143   |   0.8776   |   0.0478  |
|      11      |   0.2325  |    0.1546   |    0.1072   |   0.2327   |   0.0039  |
|      12      |   0.2047  |    0.1018   |    0.1161   |   0.5963   |   0.0332  |
|      13      |   0.2266  |    0.1002   |    0.1181   |   0.5999   |   0.0195  |
|      14      |   0.1763  |    0.1027   |    0.0938   |   0.8565   |   0.0328  |
|      15      |   0.3502  |    0.1031   |    0.1409   |   0.1384   |   0.0801  |
|      16      |   0.1577  |    0.0875   |    0.0847   |   0.4452   |   0.0187  |
|      17      |   0.1346  |    0.0790   |    0.0840   |   0.1929   |   0.0397  |
|      18      |   0.2771  |    0.2181   |    0.1018   |   0.5219   |   0.0313  |
|      19      |   0.1669  |    0.0954   |    0.0888   |   0.5356   |   0.0415  |
|      20      |   0.2918  |    0.1229   |    0.1088   |   0.2235   |   0.0538  |
|      21      |   0.2605  |    0.2068   |    0.0804   |   1.3975   |   0.0741  |
|      22      |   0.1677  |    0.1020   |    0.0893   |   0.5302   |   0.0398  |
|      23      |   0.4052  |    0.3711   |    0.0921   |   0.7812   |   0.0271  |
|      24      |   0.1683  |    0.1061   |    0.0888   |   0.7010   |   0.0539  |
|      25      |   0.2903  |    0.2224   |    0.1237   |   0.9134   |   0.0344  |
|      26      |   0.3618  |    0.2108   |    0.1210   |   1.0567   |   0.0961  |
|      27      |   0.2841  |    0.1197   |    0.1236   |   1.4198   |   0.0450  |
|      28      |   0.2152  |    0.1248   |    0.1141   |   0.7536   |   0.0233  |
|      29      |   0.3959  |    0.1437   |    0.1593   |   0.2796   |   0.1390  |
|      30      |   0.2785  |    0.1445   |    0.1417   |   2.4186   |   0.0941  |
|      31      |   0.3063  |    0.1372   |    0.1631   |   1.2989   |   0.1108  |
|      32      |   0.2315  |    0.1146   |    0.1424   |   1.5231   |   0.0545  |
|      33      |   0.4258  |    0.1672   |    0.1662   |   0.7473   |   0.2864  |
|      34      |   0.2511  |    0.1271   |    0.1455   |   1.8965   |   0.0953  |
|      35      |   0.3080  |    0.1555   |    0.1662   |   5.5289   |   0.3239  |
|      36      |   0.2871  |    0.1609   |    0.1547   |   0.7026   |   0.2108  |
|      37      |   0.3059  |    0.1623   |    0.1427   |   0.9164   |   0.0687  |
|      38      |   0.3833  |    0.1530   |    0.1381   |   0.4277   |   0.1460  |
|      39      |   0.2519  |    0.1312   |    0.1388   |   0.1946   |   0.0720  |
|      40      |   0.2327  |    0.1244   |    0.1306   |   0.4635   |   0.0954  |
|      41      |   0.1962  |    0.1128   |    0.1089   |   0.2778   |   0.0621  |
|      42      |   0.3440  |    0.1997   |    0.1027   |   0.3481   |   0.0448  |
|   Averages   |   0.2962  |    0.1877   |    0.1163   |   0.9098   |   0.0716  |
+--------------+-----------+-------------+-------------+------------+-----------+