estimate_imu_to_camera_rotation failed

[Wayne-Mai]

Hi, I'm running your examples of GoPro camera & imu calibration but I failed with following mistake. Looks like std::vector<double> cams_dt_s or timestamps_images is empty?

==================================================================
Spline weighting and knot spacing estimation took 2.31s.
==================================================================
==================================================================
Initializing IMU to camera rotation.
==================================================================
I0602 14:01:41.985631  3844 estimate_imu_to_camera_rotation.cc:72] Load IMU bias file: /home/Downloads/GoPro/GoPro9/dataset3/imu_bias/imu_bias_GH010163.json
I0602 14:01:42.436834  3844 estimate_imu_to_camera_rotation.cc:87] Using supplied initial imu 2 camera time offset: -0.078202
I0602 14:01:42.453464  3844 estimate_imu_to_camera_rotation.cc:105] Mean IMU data rate: 202.914Hz
estimate_imu_to_camera_rotation: /home/Downloads/GoPro/OpenImuCameraCalibrator/src/utils/utils.cc:78: double OpenICC::utils::MedianOfDoubleVec(std::vector<double, std::allocator<double> >&): Assertion `!double_vec.empty()' failed.
==================================================================

[urbste]

Hi, sorry for my late reply. Did you solve your issue?

[thealchemist-x]

Hi urbste,

Thank you for your wonderful work, they have been helpful for me to pick up SLAM with a gopro. My main problem right now is that after running run_gopro_calibration.py, it does not generate a pose_calib even though the system has successfully completed camera calibration.

Track Id: 62 std dev: 0.0571261 0.0681775  0.109511 mm
Mean board point standard deviation after optimization: 0.0793221 0.0805474  0.108282 mm
**I0719 11:54:17.070703 1648212 pose_estimator.cc:231] Optimizing all estimated poses.**
==================================================================
Pose estimation estimation took 30.28s.
==================================================================
==================================================================
Estimating Spline error weighting and knot spacing.
==================================================================
Knot spacing SO3:               0.040 seconds at quality level q_so3=0.99
Knot spacing  R3:               0.150 seconds at quality level q_r3=0.99
Gyroscope weighting factor:     71.911 at quality level q_so3=0.99
Accelerometer weighting factor: 7.781 at quality level q_r3=0.99
Writing result to:  /home/ICT2000/klow/Summer2022/GoPro-Hack/CalibrationDataset4/cam_imu/spline_info_GX010019.json
==================================================================
Spline weighting and knot spacing estimation took 1.44s.
==================================================================
==================================================================
Initializing IMU to camera rotation.
==================================================================
E0719 11:54:28.651844 1661022 reconstruction_reader.cc:65] Could not open the file: /home/ICT2000/klow/Summer2022/GoPro-Hack/CalibrationDataset4/cam_imu/**pose_calib_GX010019.calibdata for reading**.

On closer look, it seems that within estimate_camera_poses_from_checkerboard, after it optimizes all poses, the program does not continue with filtering bad poses as there isn't any output.

I was wondering if you have any clues that may be helpful for the situation I'm in?

Thank you so much!

Much appreciated, Kenny

[thealchemist-x]

Thanks guys! I managed to resolve my issue. Turns out that ceres cranked up. I had the Theias-False issue during the make, so I had to modify the CMakeList a little to get it working.

If anyone faces similar issue, I'm willing to help!

Cheers, Kenny

[mgilson420]

Thanks guys! I managed to resolve my issue. Turns out that ceres cranked up. I had the Theias-False issue during the make, so I had to modify the CMakeList a little to get it working.

If anyone faces similar issue, I'm willing to help!

Cheers, Kenny

How did you modify the CMakeList? I'm having a similar issue.

[pierreHaslee]

I am also having this issue, the pose_calib is not found (because not created) by the following steps of the program even though calibration and position don't show any errors.

could you give any details on how you modified the cmake to make it work?

[mgilson420]

I am also having this issue, the pose_calib is not found (because not created) by the following steps of the program even though calibration and position don't show any errors.

could you give any details on how you modified the cmake to make it work?

I was able to get the pose_calib to generate, the problem is it is empty.

Use this argument: --optimize_board_points=1

Now that it generates, I'm debugging why it is empty. So it's not fixed for me. You'll know it's empty because you'll get this error:

double OpenICC::utils::MedianOfDoubleVec(std::vector<double, std::allocator<double> >&): Assertion '!double_vec.empty()' failed.

Also, I'm using "run_smartphone_calibration.py" so if you're using another calibration you may have different results.

[thealchemist-x]

Hey hi @memesto-max, seems like we may have similar issues, is your problem also on pose_estimator.cc? It might be that the reason the vector (z_values) is empty is because there isn't any views found on pose_dataset. Were you able to perform optimization of the poses? If you could, it may mean you had sufficient views to do that, but strangely after the optimization, you do not have any views left.

Hi @pierreHaslee, these are where I edited to make OpenIMUCameraCalibrator work: (1) pyTheiaSfM/build/TheiaConfig.cmake (2) In the attached TheiaConfig.cmake, Line 229, set the path to where TheiaTargets.cmake is located in. The default path should be found in pyTheiaSfM/build/CMakeFiles/Export/share/Theia, so you might have to hardcode the path if it can't auto-detect the path. (3) OpenIMUCameraCalibrator/applications/CMakeLists.txt

You can take a look at my attachments and compare them with the default files. CMakeLists.txt TheiaConfig.txt PS: I saved TheiaConfig.cmake as TheiaConfig.txt in order to upload it.

Hope it helps!!

[mgilson420]

So I found out now that my issue is somewhat related to the pose_estimator.FilterBadPoses(); in estimate_camera_poses_from_checkerboard.cc

What happens is that all the z values are negative for the pose estimates. This causes the FilterBadPoses() function to remove all the poses from the Theia reconstruction. This can easily be fixed by changing the if statement:

if (std::abs(diff) > std::abs(median_z))

This does not fix the issue because the IMU calibration error is 2000.0+ with these poses.

I noticed that you submitted an separate issue that is about this exact problem, not sure if you've been able to resolve that or not. Currently, I'm testing out different SolvePnP methods. My thinking is that in pyTheiaSfM there was a commit in the the SolvePnP function or something regarding poses that is causing all the poses to have a negative z value.

I'm thinking this was recently broken considering you've tested the sample datasets and they fail in the same way.

[thealchemist-x]

Hi @memesto-max, yea I submitted a separate issue on this and from @urbste, we shouldn't be seeing too many negative z-value. I do hope it's a commit error somewhere in pyTheiaSfM. I haven't resolve this issue yet, but it's great to have another person looking into this. Thank you!

[mgilson420]

Update:

I ran the cv::solvePnPRansac from opencv. The results are much more what we would expect here's a plot of Theia vs OpenCV pose estimations:

(NOTE: I mislabeled Theia and OpenCV, the Theia pose estimates are in the negative z area of the graph)

Theia's SolvePnP method is outputting a negative Z translation where OpenCV is not. Solving this issue is not as simple as inverting the Z estimate from Theia's SolvePnP because the OpenCV pose estimate does not exactly line up with Theia's. I don't know if this is an issue with the pose estimate from either library or just a rotation issue.

I'm going to try to recompile an old version of pyTheiaSfM and see if I can get decent pose estimates.

[pierreHaslee]

Hi @memesto-max , even with the --optimize_board_points=1 argument the pose_calib would not generate on my side. Thank you for helping tho!

@thealchemist89 Thank you for your answer, I will try it!

Is the z problem still there after all the cmakes fixes? Because I admit to not being an expert in this particular field and I don't think I will be able to debug this anyway.

[mgilson420]

Optimizing board points won't help much because if it did generate any poses, they would be invalid anyways.

Unfortunately, changing the cmake files will not correct the negative z issues.

Reverting to an older version of pyTheiaSfM hasn't worked so far and recompiling takes forever so I'm trying to see if I can add a rotation to correct the issue.

[urbste]

Hey guys. Could you provide me a sample dataset so I can investigate your issues?

[mgilson420]

My dataset repo

in: ./imu-test-videos/board_change/

[urbste]

Can I ask why you did not use the supplied Charuco board? https://github.com/urbste/OpenImuCameraCalibrator/blob/master/resource/board.png

The first problem I see directly is that the markers are not half the size of the checkerboard square size. Here https://github.com/urbste/OpenImuCameraCalibrator/blob/master/applications/extract_board_to_json.cc#L72 I initialize the checkerboard object points using half the marker size. This is not elegant at all and would have to change that. :-D

Could you record a dataset with the supplied board?

[mgilson420]

I did not record using the supplied board because my camera is low resolution. The larger supplied board with 5X5 aruco markers was not detecting well at all.

I made the necessary modifications to detect this new board with these changes:

extract_board_to_json.cc:45

DEFINE_int32(aruco_dict,
             cv::aruco::DICT_4X4_50,
             "Aruco dictionary id.");

extract_board_to_json.cc:72 const float aruco_marker_length = FLAGS_checker_square_length_m / 1.3067f;

I can record a dataset using your board and I'll put them in my repo. I'll let you know when they're up.

EDIT:

Also one more question, what should we expect for the final alignment error for spline weighting and knot spacing estimation? Here's what I see: I20220810 09:22:50.045789 80882 imu_to_camera_rotation_estimator.cc:285] Final alignment error: 3985.16

I'm guessing it should be <1.0 but I have no idea. 3000+ definitely doesn't sound right so I believe I still have some fixes to make.

[mgilson420]

@urbste The new dataset is now uploaded to this repo. In the folder ./imu-test-videos/standard_charuco/

I really appreciate your help!

[thealchemist-x]

@memesto-max Thanks so much for your findings with opencv and theia PnP's computation. That's really insightful!!!