FilterBadPoses removes all views

[thealchemist-x]

Hi,

I'm trying to calibrate my Hero9 and have followed the instructions. However, during estimate camera poses from checkerboard, I realized that FilterBadPoses removes all the views because the difference of the z-values is always greater than median-z.

It turns out that my median-z is a value that is < 0. I tried to modify the if-condition to std::abs(diff) - std::abs(median_z) in PoseEstimator::FilterBadPoses(), but after doing so, I found that after optimizing IMU to camera calibration using spline fusion, my reprojection error for each images after imshow() is of an absurd number (>200 pixels).

I'm not too familiar with Theia, but I'm wondering if there's any way I can visualize my camera poses after camera calibration process?

Any guidance and advice will be appreciated.

Many thanks, Kenny

[urbste]

Hi, sorry for replying so late. I am currently on vacation. I did that, because the PnP solver sometimes returned a "flipped" pose, meaning that the camera pose was on the wrong side of the checkerboard. However, I am using these camera positions to initialize the VI-spline. So if too many of these poses are on the wrong side of the board (flipped z coordinate), the estimation will fail.

Still it is weird that you would get that many failed pose estimates. Are you using a Charuco or a checkerboard? Regarding the visualization: Have a look at the *.ply files that get written to the output folder. Have a look at that line. Also try turning of the flag "FLAGS_optimize_board_points".

If you do not succeed you could send me your dataset and I could have a look after I return from vacation.

Cheers Steffen

[thealchemist-x]

Hi Steffen,

Oh wow, thank you for writing during your vacation - I appreciate that. Please enjoy your vacation!

In that case, it seems that almost all of my poses are on the wrong side of the board. Thank you for this insight. Much appreciated also on your suggestions to view *.ply files. I've seen those outputs using CloudCompare and they seem reasonable - the camera calibration and estimated camera poses' (I modified std::abs(diff) - std::abs(median_z) in PoseEstimator::FilterBadPoses()) reprojection errors were also < 1 pix.

I'm using a chessboard with aruco and my data can be found here: https://drive.google.com/file/d/14YDKuBlYrKnPsJBGN-9m3o-ITU2OHgeF/view?usp=sharing

Thank you again for open sourcing your work. I hope to contribute in some ways to your repo!

Best wishes, Kenny

[urbste]

Sure be happy for contributions. :) Also bugfixes that can improve the repo are appreciated!

[thealchemist-x]

Hi Steffen,

I am suspecting perhaps there's some axis (checkerboard/imu etc) that I have, which may not be in-alignment with your convention that resulted in this issue I'm facing. Do you think this could be possible and also I like to ask, what are the axes convention you are using for the checkerboard, camera and imu?

I find it strange that the reprojection errors during camera calibration are decent (<1pix error) and yet, after doing theia::EstimateCalibratedAbsolutePose in PoseEstimator::EstimatePosePinhole, all of my pose.position's z-value are negative.

Thank you for taking time to read and in considering this issue I'm facing, I appreciate that.

Much appreciation, Kenny

[thealchemist-x]

Hi Steffen,

I discovered that I get the same issue when I ingested your open-source dataset (GoPro9). The only thing I tweaked is that because the size of views for bundle adjustment is too huge for my RAM, I reduced the duration of the cam_imu video.

The dataset can be found here: https://drive.google.com/file/d/1atJLKuK5BfO0zFh1V_afBslmdgNRVqyf/view?usp=sharing

And these are a slice of the output from estimate_camera_poses_from_checkerboard:

I0802 16:27:21.601768 782579 pose_estimator.cc:256] Removing view 530 due to large z coordinate: -0.172488 vs median z coordinate -0.165876
z-coord: -0.173308, median-z coord: -0.165876
I0802 16:27:21.601788 782579 pose_estimator.cc:256] Removing view 531 due to large z coordinate: -0.173308 vs median z coordinate -0.165876
z-coord: -0.174219, median-z coord: -0.165876
I0802 16:27:21.601807 782579 pose_estimator.cc:256] Removing view 532 due to large z coordinate: -0.174219 vs median z coordinate -0.165876
z-coord: -0.175064, median-z coord: -0.165876
I0802 16:27:21.601827 782579 pose_estimator.cc:256] Removing view 533 due to large z coordinate: -0.175064 vs median z coordinate -0.165876
z-coord: -0.175952, median-z coord: -0.165876
I0802 16:27:21.601846 782579 pose_estimator.cc:256] Removing view 534 due to large z coordinate: -0.175952 vs median z coordinate -0.165876
z-coord: -0.176992, median-z coord: -0.165876
I0802 16:27:21.601866 782579 pose_estimator.cc:256] Removing view 535 due to large z coordinate: -0.176992 vs median z coordinate -0.165876
z-coord: -0.17804, median-z coord: -0.165876
I0802 16:27:21.601886 782579 pose_estimator.cc:256] Removing view 536 due to large z coordinate: -0.17804 vs median z coordinate -0.165876
z-coord: -0.179075, median-z coord: -0.165876
I0802 16:27:21.601907 782579 pose_estimator.cc:256] Removing view 537 due to large z coordinate: -0.179075 vs median z coordinate -0.165876
z-coord: -0.180307, median-z coord: -0.165876
I0802 16:27:21.601925 782579 pose_estimator.cc:256] Removing view 538 due to large z coordinate: -0.180307 vs median z coordinate -0.165876
z-coord: -0.178222, median-z coord: -0.165876
I0802 16:27:21.601944 782579 pose_estimator.cc:256] Removing view 539 due to large z coordinate: -0.178222 vs median z coordinate -0.165876
z-coord: -0.181528, median-z coord: -0.165876
I0802 16:27:21.601970 782579 pose_estimator.cc:256] Removing view 540 due to large z coordinate: -0.181528 vs median z coordinate -0.165876
**Kenny: pose_dataset.ViewIds().size()=0**

You can see that towards the end, after filtering the bad poses, there's essentially none left for subsequent processes. This observation is the same for my dataset.

Thank you for reading and looking through my issue!

Much appreciation, Kenny

[urbste]

Hi Kenny,

I just tried your sample dataset and was able calibrate without any issues?! So this is odd. Maybe you are using some older version of TheiaSfM? Can you make sure that you are using DLS as a pose estimator in line 61 of the pose_estimator.cc file? It should look like this:

theia::PnPType pnpr = theia::PnPType::DLS;
theia::EstimateCalibratedAbsolutePose(
      ransac_params_, theia::RansacType::RANSAC, pnpr, correspondences_undist, &pose,
      &ransac_summary);

In general I looked at the IMU-CAMERA calibration video. You could probably add bit more dynamics to the movement, however, this has nothing to do with your issue. I ran it with the following line:

python python/run_gopro_calibration.py --path_calib_dataset /media/Data/Sparsenet/CameraCalibrationStudy/UserDatasets/CalibrationDatasetGopro --image_downsample_factor 1.5 --camera_model DIVISION_UNDISTORTION --verbose 1 --optimize_board_points 0

Result log:

I0810 13:48:29.612668 189840 continuous_time_imu_to_camera_calibration.cc:216] Mean reprojection error 1.34706px
I0810 13:48:29.612681 189840 continuous_time_imu_to_camera_calibration.cc:217] Mean reprojection error after line delay optim 1.34706px
g: -0.290273  -9.78191 -0.159489
accel_bias at time 0: 0.00362219  -0.113916    0.04156
gyro_bias at time 0:  -0.00212417   0.00149206 -0.000906517
T_i_c qw,qx,qy,qz: 0.00375274 0.0098266 -0.705207 0.708923
T_i_c t: 0.000955085    0.047623 -0.00282411
T_i_c R:   -0.999779  -0.0191804  0.00863968
-0.00853877 -0.00533667   -0.999949
  0.0192255   -0.999802  0.00517172
Initialized line delay [us]: 46.3426
Calibrated line delay [us]: 46.3426

[urbste]

Btw, I just installed the master branch of pyTheiaSfM and also recompiled this repo. Your dataset ran without issues.

[urbste]

I also ran it with --optimize_board_points 1:

It takes a long time but improves the reprojection error a bit:

I0810 14:27:55.394011 267089 continuous_time_imu_to_camera_calibration.cc:216] Mean reprojection error 1.07609px
I0810 14:27:55.394022 267089 continuous_time_imu_to_camera_calibration.cc:217] Mean reprojection error after line delay optim 1.07609px
g: -0.287337  -9.78202 -0.163121
accel_bias at time 0: 0.00362219  -0.113916    0.04156
gyro_bias at time 0:  -0.00212417   0.00149206 -0.000906517
T_i_c qw,qx,qy,qz: 0.00363517 0.00960425 -0.705297 0.708837
T_i_c t:   0.0010181    0.052321 -0.00312139
T_i_c R:   -0.999789  -0.0187012  0.00848795
-0.00839421 -0.00508535   -0.999952
  0.0187434   -0.999812  0.00492729
Initialized line delay [us]: 46.3426
Calibrated line delay [us]: 46.3426

[urbste]

The visualization shows, that you need to excite the accelerometer a bit more by doing faster camera motions:

[mgilson420]

Since this is definitely tied to the other issue I figured I'd run a test of this dataset (CalibrationDatasetGopro) with your parameters and I had no luck

Ubuntu version: 20.04 OpenCV+contrib version: 4.2.0 Eigen version: 3.3.7 Ceres version: 2.1.0 Pytheia version: 0.1.7 OpenImuCameraCalibrator: latest commit (080cf97676e9ea3c9fe965e9cb4bd4628fea078c)

still:

Removing view 538 due to large z coordinate: 0.0894691, 0.245561, -0.110265  vs median z coordinate -0.134001
Removing view 539 due to large z coordinate: 0.0871948, 0.246284, -0.109877  vs median z coordinate -0.134001
Removing view 540 due to large z coordinate: 0.085727, 0.244076, -0.111656  vs median z coordinate -0.134001
==================================================================
Pose estimation estimation took 38.10s.
==================================================================
==================================================================
Estimating Spline error weighting and knot spacing.
==================================================================
Knot spacing SO3:               0.040 seconds at quality level q_so3=0.99
Knot spacing  R3:               0.150 seconds at quality level q_r3=0.99
Gyroscope weighting factor:     60.529 at quality level q_so3=0.99
Accelerometer weighting factor: 6.853 at quality level q_r3=0.99
Writing result to:  ../../Downloads/CalibrationDatasetGopro/cam_imu/spline_info_GX010024.json
==================================================================
Spline weighting and knot spacing estimation took 3.41s.
==================================================================
==================================================================
Initializing IMU to camera rotation.
==================================================================
I20220810 12:29:19.746582 88297 estimate_imu_to_camera_rotation.cc:72] Load IMU bias file: ../../Downloads/CalibrationDatasetGopro/imu_bias/imu_bias_GX010011.json
I20220810 12:29:20.435760 88297 estimate_imu_to_camera_rotation.cc:90] Using supplied initial imu 2 camera time offset: -0.112136
I20220810 12:29:20.465672 88297 estimate_imu_to_camera_rotation.cc:108] Mean IMU data rate: 198.598Hz
estimate_imu_to_camera_rotation: /home/max/imucalib-newest/OpenImuCameraCalibrator/src/utils/utils.cc:78: double OpenICC::utils::MedianOfDoubleVec(std::vector<double, std::allocator<double> >&): Assertion `!double_vec.empty()' failed.
==================================================================
Spline weighting and knot spacing estimation took 1.35s.
==================================================================

Sounds like your current setup is a tad different than what we're running, otherwise I have no clue why our results are so different

[mgilson420]

@urbste Are you using the master branches of both pyTheiaSfM and OpenImuCameraCalibrator?

I noticed there are some branches that are have commits ahead of the master branches. Also, from what I can tell the master branch of pyTheiaSfM is using TheiaSfM v0.7 NOT v0.8.

Would this be a problem?