Error in `rosrun atom_calibration calibrate`

[masakiyamamotobb]

In my test to check whether I can use ATOM framework, I've stuck in an error. I'll appreciate it very much if you point out any of my mistakes.

root@nrp:~/nrp/catkin_ws# rosrun atom_calibration calibrate -json /root/nrp/catkin_ws/src/my_calib/output/dataset.json
Skipped loading images and point clouds for collections: [].
Deleted collections: ['001'] because these are incomplete. If you want to use them set the use_incomplete_collections flag.
Deleted collections: []: at least one detection by a camera should be present.
After filtering, will use 6 collections: ['000', '002', '003', '004', '005', '006']
Loaded dataset containing 2 sensors and 6 collections.
Selected collection key is 000

Initializing optimizer...
Anchored sensor is camera1
Creating parameters ...
Creating residuals ... 
RNG Seed: 3112602151475238098
Computing sparse matrix ... 
Initializing optimization ...
One optimizer iteration has 13 function calls.
Starting optimization ...
   Iteration     Total nfev        Cost      Cost reduction    Step norm     Optimality   
       0              1         1.8323e+05                                    8.67e+05    
       1              2         1.6510e+05      1.81e+04       7.07e-02       7.98e+05    
       2              3         1.6402e+05      1.08e+03       1.09e-02       7.91e+05    
       3              4         1.3945e+05      2.46e+04       1.17e-01       7.08e+05    
       4              5         1.2507e+05      1.44e+04       7.06e-02       6.59e+05    
       5              6         1.1412e+05      1.10e+04       5.39e-02       6.21e+05    
       6              7         1.1213e+05      1.99e+03       9.35e-03       6.14e+05    
       7              8         9.8624e+04      1.35e+04       6.74e-02       5.66e+05    
       8              9         8.2557e+04      1.61e+04       8.04e-02       5.06e+05    
       9             10         7.8191e+04      4.37e+03       2.01e-02       4.89e+05    
      10             11         5.4925e+04      2.33e+04       1.23e-01       3.92e+05    
      11             12         4.1209e+04      1.37e+04       8.24e-02       3.28e+05    
      12             13         4.1031e+04      1.79e+02       1.33e-02       3.24e+05    
      13             14         4.0854e+04      1.77e+02       1.31e-02       3.20e+05    
      14             15         4.0681e+04      1.73e+02       1.27e-02       3.16e+05    
      15             16         4.0001e+04      6.80e+02       7.81e-03       3.13e+05    
      16             17         2.1259e+04      1.87e+04       5.79e-01       2.14e+05    
      17             18         2.0756e+04      5.03e+02       3.54e-02       2.08e+05    
      18             19         2.0509e+04      2.47e+02       2.35e-02       1.97e+05    
      19             20         1.6334e+04      4.17e+03       3.35e-01       1.55e+05    
      20             21         1.6188e+04      1.46e+02       1.96e-02       1.47e+05    
      21             22         1.6053e+04      1.35e+02       1.96e-02       1.39e+05    
      22             23         1.3041e+04      3.01e+03       3.39e-01       1.07e+05    
      23             24         1.2910e+04      1.31e+02       1.76e-02       9.25e+04    
      24             25         1.2822e+04      8.77e+01       1.50e-02       8.12e+04    
      25             26         1.2760e+04      6.20e+01       1.31e-02       7.22e+04    
      26             27         1.2714e+04      4.61e+01       1.15e-02       6.50e+04    
      27             28         1.2677e+04      3.69e+01       1.03e-02       5.94e+04    
      28             29         1.2646e+04      3.08e+01       9.23e-03       5.50e+04    
      29             30         1.2619e+04      2.73e+01       8.43e-03       5.15e+04    
      30             31         1.2594e+04      2.51e+01       7.86e-03       4.87e+04    
      31             32         1.2570e+04      2.37e+01       7.41e-03       4.64e+04    
      32             34         1.2355e+04      2.15e+02       2.50e-02       3.23e+04    
      33             35         1.2106e+04      2.49e+02       1.80e-01       1.65e+04    
      34             37         1.2059e+04      4.71e+01       5.10e-02       1.77e+04    
      35             38         1.2041e+04      1.77e+01       3.69e-02       9.24e+03    
      36             40         1.2034e+04      7.71e+00       1.82e-03       5.51e+03    
      37             41         1.2026e+04      7.82e+00       1.37e-02       5.82e+03    
      38             42         1.2022e+04      3.62e+00       2.54e-02       4.23e+03    
      39             44         1.2020e+04      1.91e+00       2.45e-03       2.88e+03    
      40             45         1.2019e+04      9.52e-01       7.81e-04       2.08e+03    
`ftol` termination condition is satisfied.
Function evaluations 45, initial cost 1.8323e+05, final cost 1.2019e+04, first-order optimality 2.08e+03.

-----------------------------
Optimization finished in 1.59576 secs: `ftol` termination condition is satisfied.
Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+--------------+--------------+
| Collection | camera1 (px) | camera2 (px) |
+------------+--------------+--------------+
|    000     |   25.4278    |   28.7574    |
|    002     |   21.7988    |   79.8011    |
|    003     |   12.1166    |   40.0449    |
|    004     |   94.4420    |   111.1386   |
|    005     |   31.3031    |   59.8192    |
|    006     |   76.1379    |   71.4401    |
|  Averages  |   43.5377    |   65.1669    |
+------------+--------------+--------------+
output_folder is: /root/nrp/catkin_ws/src/my_calib/output
Saved json output file to /root/nrp/catkin_ws/src/my_calib/output/atom_calibration.json.
Traceback (most recent call last):
  File "/root/nrp/catkin_ws/devel/lib/atom_calibration/calibrate", line 15, in <module>
    exec(compile(fh.read(), python_script, 'exec'), context)
  File "/root/nrp/catkin_ws/src/3rdparty/atom/atom_calibration/scripts/calibrate", line 469, in <module>
    main()
  File "/root/nrp/catkin_ws/src/3rdparty/atom/atom_calibration/scripts/calibrate", line 465, in main
    saveResultsXacro(dataset, selected_collection_key)
  File "/root/nrp/catkin_ws/src/3rdparty/atom/atom_core/src/atom_core/xacro_io.py", line 52, in saveResultsXacro
    joint.origin.xyz = trans
AttributeError: 'NoneType' object has no attribute 'xyz'

This is the simplest test I can think of and its urdf and summary file is attached.

summary.pdf two_cameras.urdf.txt

[miguelriemoliveira]

Hi @masakiyamamotobb ,

Thanks. Is the link correct? The bagfile's name in the link you sent is data_2023-04-14-08-40-36.bag, which is the name of the bag I had before ...

[yamamotomas]

I'm sorry. This will work:

https://drive.google.com/file/d/1UIIYgOyZhswK1UHJSKbnRwQNO1NWuOtV/view?usp=share_link

[yamamotomas]

Finally I have calibrated two cameras with calib.io's software (https://calib.io/products/calib).

In conclusion, ros camera_info and calibration results are mostly the same. So I'm going to leave them as it is for a while.

Camera1:

/camera1/color/camera_info
height: 480
width: 640
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [613.9519653320312, 0.0, 321.2705993652344, 0.0, [613.6005859375](tel:6136005859375), 233.10227966308594, 0.0, 0.0, 1.0]

calib.io result
height: 480
width: 640
D: [0.15355816666139743, -0.3289751812254585, -6.328734849366592e-05, 0.00015164782181395655, -0.042770691682211884]
K: [604.8215142379465, 0.0, 323.35292396683775, 0.0, 604.8215142379465, 231.91286204245972, 0.0, 0.0, 1.0]

Camera2:

/camera2/color/camera_info
height: 480
width: 640
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [607.23583984375, 0.0, 321.7033386230469, 0.0, 606.1530151367188, 233.6876678466797, 0.0, 0.0, 1.0]

calib.io result
height: 480
width: 640
D: [0.13463235580391597, -0.3227427564493204, 0.0010043320023022865, -0.0010660306530842795, 0.08298263852941928]
K: [599.9799799783843, 0.0, 321.2926259554301, 0.0, 599.9799799783843, 233.4189579460085, 0.0, 0.0, 1.0]

[miguelriemoliveira]

Hi @yamamotomas ,

In conclusion, ros camera_info and calibration results are mostly the same. So I'm going to leave them as it is for a while.

Right, as we expected that should not be the problem.

About the new bagfile. I created a dataset and looked into the images and the desynchronization appears to be minimal.

at most I've seem a couple of hundreds milliseconds of difference, so this bagfile is quite nice. Let me see how to calibration runs on this.

[miguelriemoliveira]

Hi @danifpdra ,

can you please share the mmtbot datasets we used for the ATOM paper? I wanted to compared against this system to try to find whats wrong.

[miguelriemoliveira]

Using a single collection and both cameras, there is a strange solution where the system places one camera on top of the other ... this is the result of the optimization

[miguelriemoliveira]

This is the tf tree

So the way the links are organized in the nrp_robot is strange:

In particular, the camera_links are usually with x pointing forward, in this case they are like the optical axis, z point forward. But even stranger is that the position of the link1 and link2 are on top of each other, and not side by side.

In any case this should not be a problem for ATOM. Will keep searching...

[miguelriemoliveira]

Hi @masakiyamamotobb ,

At last, I found the problem!!!!. In the config.yaml, the pattern is set to fixed.

Indeed, the pattern is not moving. But you are moving the camera's instead of the pattern, and that's the equivalent of having the sensors fixed and a moving pattern, which is different for all collections. So you must set the pattern to dynamic.

We use fixed patterns when the sensor moves, e.g., a hand eye calibration.

Now it calibrates well for several collections:

+------------+--------------+--------------+
| Collection | camera1 (px) | camera2 (px) |
+------------+--------------+--------------+
|    000     |    0.6360    |    0.5058    |
|    001     |    0.3268    |    0.6184    |
|    002     |    0.4101    |    0.6119    |
|    003     |    0.6498    |    0.5225    |
|  Averages  |    0.5057    |    0.5646    |
+------------+--------------+--------------+

This was a hard one because the inconsistency between collections led me to think this was a problem with synchronization, then some strange bug.

Very happy to know ATOM is still up and running.

I am comiting the changes to the nrp git repo.

[danifpdra]

Hi @danifpdra ,

can you please share the mmtbot datasets we used for the ATOM paper? I wanted to compared against this system to try to find whats wrong.

dataset_31_03_2021.zip

[miguelriemoliveira]

Hi @danifpdra , can you please share the mmtbot datasets we used for the ATOM paper? I wanted to compared against this system to try to find whats wrong.

dataset_31_03_2021.zip

Thanks @danifpdra .

[yamamotomas]

@miguelriemoliveira Great thanks for your efforts !!! I have also recreated the same result. I'll start to apply ATOM to our robot system! https://www.tandfonline.com/doi/full/10.1080/01691864.2022.2109429

+------------+--------------+--------------+
| Collection | camera1 (px) | camera2 (px) |
+------------+--------------+--------------+
|    000     |    0.2262    |    0.2083    |
|    001     |    0.2349    |    0.2753    |
|    002     |    0.4296    |    0.3468    |
|    003     |    0.2597    |    0.2911    |
|    004     |    0.2441    |    0.2161    |
|  Averages  |    0.2789    |    0.2675    |
+------------+--------------+--------------+

[miguelriemoliveira]

Ok. I will close this issue.

If you need additional help, we have an system put in place where we give two workshops on Atom, one theorethical and the other more practical, and then provide support in the calibration of your robotic system. If you are interested let me know.

Best, Miguel