Too many measurements at collect data step

[sensorCalib]

Hi,

at step 4. when trying to collect data, we seem to get a dimension mismatch. This is the error occuring:

[ERROR] [1656064461.734798, 1656052507.736720]: bad callback: <bound method InteractiveDataLabeler.sensorDataReceivedCallback of <atom_calibration.collect.interactive_data_labeler.InteractiveDataLabeler object at 0x7f3929d15d60>> Traceback (most recent call last): File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/home/user/ws/src/atom-calibration/atom_calibration/src/atom_calibration/collect/interactive_data_labeler.py", line 295, in sensorDataReceivedCallback self.labelData() # label the data File "/home/user/ws/src/atom-calibration/atom_calibration/src/atom_calibration/collect/interactive_data_labeler.py", line 464, in labelData self.labels, seed_point, inliers = labelPointCloud2Msg(self.msg, x_marker, y_marker, z_marker, File "/home/user/ws/src/atom-calibration/atom_calibration/src/atom_calibration/collect/label_messages.py", line 45, in labelPointCloud2Msg points[:, 0] = pc['x'].flatten() # flatten because some pcs are of shape (npoints,1) rather than (npoints,) ValueError: could not broadcast input array from shape (65536) into shape (64)

We have 64 lines with 1024 measurement-values each, which results in the 65536 measurements. However, only one measurment per coordinate is expected.

Any idea how to fix this? Thank you very much.

[miguelriemoliveira]

2. Install a special fork of rviz as explained here

Just thought about it ... I don't think you'll need this since you don't have a depth camera.

[sensorCalib]

So I am really lost at why you had that problem. The agri-gaia/launch/playbag.launch should set the parameter to true ... Are you using this launch file (using the dataset_playback.launch calls this one)?

I called

roslaunch agri_gaia_calibration dataset_playback.launch

so I guess that I am using "agri-gaia/launch/playbag.launch" (?).

As for the calibration: I corrected the labels and started a calibration, these were the results:

Initializing optimization ... One optimizer iteration has 13 function calls. Starting optimization ... Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0557 | 0.3102 | | 001 | 0.0409 | 157.5191 | | 002 | 0.0408 | 8.8233 | | 003 | 0.0394 | 99.9691 | | 004 | 0.0346 | 133.7405 | | 005 | 0.0616 | 69.8063 | | Averages | 0.0455 | 78.3614 | +------------+------------+----------+ Iteration Total nfev Cost Cost reduction Step norm Optimality
0 1 1.0620e+05 1.29e+05
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0303 | 26.5169 | | 001 | 0.0303 | 133.0785 | | 002 | 0.0203 | 27.6154 | | 003 | 0.0256 | 124.7527 | | 004 | 0.0246 | 108.7841 | | 005 | 0.0323 | 95.5582 | | Averages | 0.0272 | 86.0510 | +------------+------------+----------+ 1 2 1.0273e+05 3.47e+03 3.45e-01 3.74e+04
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0272 | 23.6644 | | 001 | 0.0341 | 135.4535 | | 002 | 0.0191 | 29.1886 | | 003 | 0.0230 | 122.7202 | | 004 | 0.0284 | 112.1459 | | 005 | 0.0313 | 91.2909 | | Averages | 0.0272 | 85.7439 | +------------+------------+----------+ 2 5 1.0270e+05 3.28e+01 5.55e-02 5.32e+04
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0297 | 22.6901 | | 001 | 0.0312 | 135.3852 | | 002 | 0.0190 | 27.2675 | | 003 | 0.0244 | 122.3284 | | 004 | 0.0254 | 111.7949 | | 005 | 0.0328 | 91.4350 | | Averages | 0.0271 | 85.1502 | +------------+------------+----------+ 3 6 1.0211e+05 5.91e+02 1.26e-02 2.94e+04
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0347 | 22.4792 | | 001 | 0.0266 | 135.8605 | | 002 | 0.0193 | 24.6667 | | 003 | 0.0271 | 121.6614 | | 004 | 0.0213 | 111.7909 | | 005 | 0.0360 | 91.9589 | | Averages | 0.0275 | 84.7363 | +------------+------------+----------+ 4 7 1.0195e+05 1.60e+02 2.49e-02 1.76e+04
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0341 | 21.7835 | | 001 | 0.0271 | 136.1708 | | 002 | 0.0192 | 24.6185 | | 003 | 0.0267 | 121.3022 | | 004 | 0.0216 | 112.2154 | | 005 | 0.0357 | 91.3073 | | Averages | 0.0274 | 84.5663 | +------------+------------+----------+ 5 8 1.0183e+05 1.18e+02 7.17e-03 9.13e+03
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0341 | 21.1330 | | 001 | 0.0271 | 136.6238 | | 002 | 0.0193 | 24.5644 | | 003 | 0.0266 | 120.8531 | | 004 | 0.0216 | 112.7777 | | 005 | 0.0357 | 90.6029 | | Averages | 0.0274 | 84.4258 | +------------+------------+----------+ 6 9 1.0179e+05 3.98e+01 7.18e-03 4.78e+03
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0353 | 20.8052 | | 001 | 0.0263 | 137.0294 | | 002 | 0.0199 | 23.8438 | | 003 | 0.0273 | 120.4277 | | 004 | 0.0211 | 113.1125 | | 005 | 0.0362 | 90.3797 | | Averages | 0.0277 | 84.2664 | +------------+------------+----------+ 7 10 1.0177e+05 1.53e+01 9.58e-03 4.24e+03
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0346 | 20.4293 | | 001 | 0.0269 | 137.2521 | | 002 | 0.0202 | 24.1191 | | 003 | 0.0271 | 120.2286 | | 004 | 0.0216 | 113.4497 | | 005 | 0.0355 | 89.8991 | | Averages | 0.0276 | 84.2297 | +------------+------------+----------+ 8 11 1.0177e+05 2.01e+00 7.12e-03 6.85e+03
Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0348 | 20.4498 | | 001 | 0.0267 | 137.2471 | | 002 | 0.0203 | 23.9775 | | 003 | 0.0272 | 120.2209 | | 004 | 0.0216 | 113.4144 | | 005 | 0.0356 | 89.9654 | | Averages | 0.0277 | 84.2125 | +------------+------------+----------+ 9 12 1.0176e+05 9.96e+00 2.14e-03 3.92e+03
ftol termination condition is satisfied. Function evaluations 12, initial cost 1.0620e+05, final cost 1.0176e+05, first-order optimality 3.92e+03.

---

Optimization finished in 31.70428 secs: ftol termination condition is satisfied. Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+----------+ | Collection | ouster_os1 | zed2i_0 | +------------+------------+----------+ | 000 | 0.0348 | 20.4498 | | 001 | 0.0267 | 137.2471 | | 002 | 0.0203 | 23.9775 | | 003 | 0.0272 | 120.2209 | | 004 | 0.0216 | 113.4144 | | 005 | 0.0356 | 89.9654 | | Averages | 0.0277 | 84.2125 | +------------+------------+----------+ output_folder is: /home/martelv/catkin_ws/data/ag_out2 Saved json output file to /home/martelv/catkin_ws/data/ag_out2/atom_calibration.json.

ag_out2.zip

The average error is higher than in your case, but I don't know how to interpret these numbers. Could you explain what they mean and when they are small enough?

[miguelriemoliveira]

Hi @sensorCalib ,

About the use_sim_time, I figured it out. See #527.

About the calibration, camera error is in pixels, lidar in meters (opened #528 to improve that).

I will try to pick it up tonight or tomorrow and get back to you.

Have a nice weekend.

[miguelriemoliveira]

Hi again @sensorCalib,

so I looked around for a reason why the errors were so high and there is a problem in the calibration.yaml.

We are saying that the calibration pattern is fixed (line 114). In your case the pattern is not fixed, since it moves from collection to collection.

I will fix this and results should be better.

[miguelriemoliveira]

Hi again @sensorCalib ,

Optimization finished in 4.20546 secs: `xtol` termination condition is satisfied.
Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+------------+---------+
| Collection | ouster_os1 | zed2i_0 |
+------------+------------+---------+
|    000     |   0.0073   |  0.4478 |
|    001     |   0.0071   |  0.3347 |
|    002     |   0.0078   |  0.3116 |
|    003     |   0.0064   |  0.4439 |
|    004     |   0.0075   |  0.4209 |
|  Averages  |   0.0072   |  0.3918 |
+------------+------------+---------+

So now its much better. Camera errors under 1 pixel.

Can you try to achieve these values as well?

Note: I had to manually change the parameter fixed from true to false. For consistency I did it in both the dataset.json and the dataset_corrected.json files. But this was a hack to avoid having to create a new dataset.

If you change the calibration.yaml and collect a new dataset it should be fine.

[miguelriemoliveira]

I ran the point_cloud_to_image evaluation to see if the calibration makes sense:

clear && rosrun atom_evaluation point_cloud_to_image -json atom_calibration.json -ls ouster_os1 -cs zed2i_0

Looks good. Color transitions are changes in lidar range measurements, and are aligned with the depth changes in the image. So the calibration looks good, even though it should contain around 20 or 30 collections instead of just 5.

[sensorCalib]

Hi again @sensorCalib ,

Optimization finished in 4.20546 secs: `xtol` termination condition is satisfied.
Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+------------+---------+
| Collection | ouster_os1 | zed2i_0 |
+------------+------------+---------+
|    000     |   0.0073   |  0.4478 |
|    001     |   0.0071   |  0.3347 |
|    002     |   0.0078   |  0.3116 |
|    003     |   0.0064   |  0.4439 |
|    004     |   0.0075   |  0.4209 |
|  Averages  |   0.0072   |  0.3918 |
+------------+------------+---------+

So now its much better. Camera errors under 1 pixel.

Can you try to achieve these values as well?

Note: I had to manually change the parameter fixed from true to false. For consistency I did it in both the dataset.json and the dataset_corrected.json files. But this was a hack to avoid having to create a new dataset.

If you change the calibration.yaml and collect a new dataset it should be fine.

I used your quick hack too and modified the parameter "fixed" from true to false, this was the result:

Optimization finished in 29.78579 secs: `xtol` termination condition is satisfied.
Errors per collection (anchored sensor,  max error per sensor, not detected as "---")
+------------+----------------+--------------+
| Collection | ouster_os1 (m) | zed2i_0 (px) |
+------------+----------------+--------------+
|    000     |     0.0241     |    0.3103    |
|    001     |     0.0120     |    0.3321    |
|    002     |     0.0261     |    0.3117    |
|    003     |     0.0235     |    0.4718    |
|    004     |     0.0415     |    0.3959    |
|    005     |     0.0324     |    0.3241    |
|  Averages  |     0.0266     |    0.3576    |
+------------+----------------+--------------+

Hi @miguelriemoliveira

thank your very much again for these fixes and adjustments.

After evaluating the calibration with

clear && rosrun atom_evaluation point_cloud_to_image -json atom_calibration.json -ls ouster_os1 -cs zed2i_0

I got this result:

This looks off.

[miguelriemoliveira]

Hi ,

Can you share your dataset? I think your labelling may have something wrong ...

On Tue, Aug 16, 2022, 09:06 sensorCalib @.***> wrote:

Hi again @sensorCalib https://github.com/sensorCalib ,

Optimization finished in 4.20546 secs: xtol termination condition is satisfied. Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+------------+---------+| Collection | ouster_os1 | zed2i_0 | +------------+------------+---------+| 000 | 0.0073 | 0.4478 || 001 | 0.0071 | 0.3347 || 002 | 0.0078 | 0.3116 || 003 | 0.0064 | 0.4439 || 004 | 0.0075 | 0.4209 || Averages | 0.0072 | 0.3918 | +------------+------------+---------+

So now its much better. Camera errors under 1 pixel.

Can you try to achieve these values as well?

Note: I had to manually change the parameter fixed from true to false. For consistency I did it in both the dataset.json and the dataset_corrected.json files. But this was a hack to avoid having to create a new dataset.

If you change the calibration.yaml and collect a new dataset it should be fine.

I used your quick hack too and modified the parameter "fixed" from true to false, this was the result:

Optimization finished in 29.78579 secs: xtol termination condition is satisfied. Errors per collection (anchored sensor, max error per sensor, not detected as "---") +------------+----------------+--------------+| Collection | ouster_os1 (m) | zed2i_0 (px) | +------------+----------------+--------------+| 000 | 0.0241 | 0.3103 || 001 | 0.0120 | 0.3321 || 002 | 0.0261 | 0.3117 || 003 | 0.0235 | 0.4718 || 004 | 0.0415 | 0.3959 || 005 | 0.0324 | 0.3241 || Averages | 0.0266 | 0.3576 | +------------+----------------+--------------+

After evaluating the calibration with

clear && rosrun atom_evaluation point_cloud_to_image -json atom_calibration.json -ls ouster_os1 -cs zed2i_0

I got this result: [image: grafik] https://user-images.githubusercontent.com/108126721/184828785-42f71c0b-b24e-42f5-a87b-4e40da1f8872.png

This looks off.

— Reply to this email directly, view it on GitHub https://github.com/lardemua/atom/issues/498#issuecomment-1216288604, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVQRJ7AJU6WHM6X4LA3VZND2HANCNFSM5ZXQDXUQ . You are receiving this because you were mentioned.Message ID: @.***>

[sensorCalib]

Here is the dataset I used: ag_out2.zip

[miguelriemoliveira]

Thanks @sensorCalib ,

I have done several experiments and I get the same results. The alignment is not the best,

I did not understand whats the problem yet. Strange because in my dataset it calibrates fine, Can you try with my dataset?

I will try to investigate further.

[sensorCalib]

The calibration looks fine with your dataset:

(In a previous and now deleted message, I assumed that I used the wrong mesh-file for the chessboard pattern, but I am not sure. Is "package://atom_calibration/meshes/chessboard_7x11_corners/pattern.dae" (the mesh-file used in my dataset ag_out2) equivalent to "package://agri_gaia_description/meshes/pattern_chess.dae" (the mesh-file used in your dataset test1) ? The latter has been removed by you some commits ago, but can be found here https://github.com/miguelriemoliveira/agri-gaia-test/blob/f5bb191a8abd84c2f1ff7cf71fc6dea37e2214a9/agri_gaia_description/meshes/pattern_chess.dae )

[miguelriemoliveira]

That's when I changed the pattern to have the correct reference frame and added it to the atom calibration package so other could use it as well.

Its here:

https://github.com/lardemua/atom/tree/noetic-devel/atom_calibration/meshes/chessboard_7x11_corners

But you just have to git pull the atom repo and then the mesh should be automatically found.

The "package://atom_calibration/meshes/chessboard_7x11_corners/pattern.dae" will resolve to the path of the atom_calibration package to each system, i.e. in mine its:

/home/mike/catkin_ws/src/calibration/atom/atom_calibration/meshes/chessboard_7x11_corners/pattern.dae

in your system it should be something else.

[miguelriemoliveira]

In any case this mesh issue should not be the problem, because its only used for drawing the mesh. I think your dataset has something wrong, although I cannot understand what ... can you please try to produce a new dataset (making sure you have the updated calibration.yaml and reconfigured your calibration package) and see if that one works?

[miguelriemoliveira]

The strange thing about your dataset is that each collection separately calibrates very well, e.g. for collection 5:

But whenever I try to use more than one collection, it seems the system gets confused.

For example, using only collections 4 and 5

Its as if the camera and the lidar change positions between each collection. How is the lidar and camera synchronization? Its all in the same computer? Could it be that the image and lidar data is somehow desynchronized so that the pattern is on the left in the image, and on the right on the lidar?

This is confusing ...

[miguelriemoliveira]

Hi @sensorCalib ,

I did not have any ideas on how to solve your mystery dataset. I will wait that you confirm this also occurs for another dataset before trying anything else ok?

Thanks, Miguel

[sensorCalib]

Hi @miguelriemoliveira ,

thank you for trying to find the issue with my dataset. I will give it another try later this week.

[miguelriemoliveira]

Ok, great. Let me know if I can help.

[sensorCalib]

Its as if the camera and the lidar change positions between each collection. How is the lidar and camera synchronization? Its all in the same computer? Could it be that the image and lidar data is somehow desynchronized so that the pattern is on the left in the image, and on the right on the lidar?

This is confusing ...

Hi @miguelriemoliveira , it turns out your assumption is correct: lidar and camera are running on the same computer, but they are not synchronized. I guess in some of the images from my dataset, there had been a movement that was too fast so that camera and lidar were off.

So, to get around this problem, I should either try to synchronize the sensor data or record another bagfile in which the pattern is moving at a super low speed (?)

[miguelriemoliveira]

Hi @sensorCalib ,

I will write some directives on how to take a good bag file and put them in the documentation. Will get back to you when its online ...

[miguelriemoliveira]

Hi @sensorCalib ,

so I added some guidelines on how to record a good bag file for calibration. My suggestion is that you read this section again

https://lardemua.github.io/atom_documentation/procedures/#collect-data

Let me know if something is not clear afterwards.

Thanks for the help. Miguel

[sensorCalib]

Hi @miguelriemoliveira ,

thank you very much for the additional documentation. I have another basic question: Before calibration, the human estimated coordinates for the Zed are https://github.com/miguelriemoliveira/agri-gaia-test/blob/main/agri_gaia_description/urdf/agri_gaia.urdf.xacro#L46 and for the Ouster they are https://github.com/miguelriemoliveira/agri-gaia-test/blob/main/agri_gaia_description/urdf/agri_gaia.urdf.xacro#L36

These human estimations shall be replaced with the results of the ATOM calibration. To directly get these distances, I modified the config-file so that the distance between the base link and the zed_base_link will be calibrated: https://github.com/miguelriemoliveira/agri-gaia-test/blob/main/agri_gaia_calibration/calibration/config.yml#L83-L84

I have run a ATOM calibration and I got the following coordinates for the Ouster https://github.com/miguelriemoliveira/agri-gaia-test/blob/main/agri_gaia_description/urdf/optimized.urdf.xacro#L64 and these for the Zed: https://github.com/miguelriemoliveira/agri-gaia-test/blob/main/agri_gaia_description/urdf/optimized.urdf.xacro#L102

At first glance, the human estimated coordinates and the ATOM calibrated coordinates for the Zed are accurate. Whereas the human estimated coordinates and the ATOM calibrated coordinates for the ouster are slightly off.

However, the ATOM calibration gives us the relative position of both sensors to each other. So, we actually only know the relative position of these sensors to each other. (Maybe the human estimation of the Ouster was corect and the human estimation of the Zed is off?)

If I have seen this right, there is no fixed point that both sensors can relate to. So, maybe it would make sense to define one of the sensors as "fixed origin"(?) Or what do you think?

Thank you very much.

[miguelriemoliveira]

Hi @sensorCalib ,

yes, we often have this problem. ATOM only estimates the relative transformation between the sensors.

The most common solution is to have one sensor fixed (we call it anchored in ATOM) and during the optimization only the other sensors can move around.

For example, if you add to config.yaml

https://github.com/miguelriemoliveira/agri-gaia-test/blob/086527616869106a0222ab099969ad110cfcc676/agri_gaia_calibration/calibration/config.yml#L152-L155

the ouster as a fixed sensor, i.e.

anchored_sensor: "ouster_os1"

then the lidar coordinates would not be touched, and only the camera's pose would be estimated.

I agree this configuration would be better. Its what we normally do in ATOM.

In theory, changing the config.yaml would required starting the process all over again, i.e. getting a new dataset etc. The problem is this takes a long time. But I think as a shortcut you can just edit the dataset.json that is given to the calibration to include this information.

Notice that the manual transformation from your robot base_link to the anchored sensor is given manually and unchanged, so you should try to measure it more or less accurately.

[miguelriemoliveira]

Hi @sensorCalib ,

I will close the issue since we had no further contact for a while. I hope ATOM worked for you, and would appreciate some feedback.

Thanks, Miguel

[sensorCalib]

Hi @miguelriemoliveira as you suggested, we fixed the ouster lidar and calibrated the camera position. The resulting calibaration file looked alright.

Thank you very much again for your detailed and extensive help.

[miguelriemoliveira]

Hi @sensorCalib ,

great news. Thanks for the feedback.