Various files missing when calibrating.

[ajsnyde]

The corners are extracted, lots of stuff is generated, but I see quite a few errors towards the end.

Command within the docker container is: python3 python/run_gopro_calibration.py --path_calib_dataset ../dataset/calibrationDataset/ --checker_size_m=0.05115 --path_to_build ../OpenImuCameraCalibrator/build/applications/

Command to start up the docker container: sudo docker run -it --rm -vpwd:/home -v /home/Downloads/GoPro9:/dataset openicc

[ajsnyde]

The MP4s are from a Hero11, for what that's worth.

[urbste]

Hi, can you share your dataset so I can have a look?

[anaidasdfg]

Same problem here! I tried it with the given videos for GoPro 9 examples and had the same issue.

[ajsnyde]

https://user-images.githubusercontent.com/11302823/222992366-b260e63d-2044-4e70-bd1c-d87c093187bf.mp4

Here's an mp4 I'm trying to calibrate with. I know, not ideal calibration material, but this is the one under 10mb.

[urbste]

@anaidasdfg Sorry, there was the Dockerfile was missing an npm install in the javascript folder. I pushed an update please check it out :)

[urbste]

@ajsnyde Is that the original file from your GoPro? It does not seem to contain a metadata stream:

I am trying to extract the metadata with the script *extract_telemetry_from_folder.py python python/extract_telemetry_from_folder.py --path ../dataset/GoProAjsnyde But get the following error:

Read stream data from 222992366-b260e63d-2044-4e70-bd1c-d87c093187bf.MP4 An error occurred Track not found Flushing on end after 10345190 bytes

[ajsnyde]

Actually, I screwed up, that's not the file I was using to calibrate the IMU. I've got a dataset but it's like 2GB in size, and can't upload it

[urbste]

Btw when I see your underwater imagery, I am not sure if you can directly use the toolbox without modification. One probably needs to adjust some stuff according to that paper

[ajsnyde]

Error! Could not find cam calibration video file with MP4 ending in path ../dataset/calibrationDataset/cam
root@7ce3ee00fab5:/OpenImuCameraCalibrator# python3 python/run_gopro_calibration.py --path_calib_dataset ../dataset/ --checker_size_m=0.05115 --path_to_build ../OpenImuCameraCalibrator/build/applications/
['../dataset/cam/GX010027.MP4']
==================================================================
Running corner extraction.
==================================================================
Extracing corners for camera calibration.
I0306 14:04:39.909039    78 extract_board_to_json.cc:92] Starting board extraction. This might take a while...
Total number of frames: 4062
I0306 14:04:41.944630    78 board_extractor.cc:443] Extracting corners from frame 60 / 4062
I0306 14:04:43.780072    78 board_extractor.cc:443] Extracting corners from frame 120 / 4062
I0306 14:04:45.552951    78 board_extractor.cc:443] Extracting corners from frame 180 / 4062
I0306 14:04:47.428068    78 board_extractor.cc:443] Extracting corners from frame 240 / 4062
I0306 14:04:49.350881    78 board_extractor.cc:443] Extracting corners from frame 300 / 4062
I0306 14:04:51.299734    78 board_extractor.cc:443] Extracting corners from frame 360 / 4062
I0306 14:04:53.173480    78 board_extractor.cc:443] Extracting corners from frame 420 / 4062
I0306 14:04:55.114609    78 board_extractor.cc:443] Extracting corners from frame 480 / 4062
I0306 14:04:57.007761    78 board_extractor.cc:443] Extracting corners from frame 540 / 4062
I0306 14:04:58.913098    78 board_extractor.cc:443] Extracting corners from frame 600 / 4062
I0306 14:05:01.036238    78 board_extractor.cc:443] Extracting corners from frame 660 / 4062
I0306 14:05:03.129483    78 board_extractor.cc:443] Extracting corners from frame 720 / 4062
I0306 14:05:04.913065    78 board_extractor.cc:443] Extracting corners from frame 780 / 4062
I0306 14:05:06.676331    78 board_extractor.cc:443] Extracting corners from frame 840 / 4062
I0306 14:05:08.450070    78 board_extractor.cc:443] Extracting corners from frame 900 / 4062
I0306 14:05:10.230935    78 board_extractor.cc:443] Extracting corners from frame 960 / 4062
I0306 14:05:12.069403    78 board_extractor.cc:443] Extracting corners from frame 1020 / 4062
I0306 14:05:13.939038    78 board_extractor.cc:443] Extracting corners from frame 1080 / 4062
I0306 14:05:15.853615    78 board_extractor.cc:443] Extracting corners from frame 1140 / 4062
I0306 14:05:17.852705    78 board_extractor.cc:443] Extracting corners from frame 1200 / 4062
I0306 14:05:19.950106    78 board_extractor.cc:443] Extracting corners from frame 1260 / 4062
I0306 14:05:22.104259    78 board_extractor.cc:443] Extracting corners from frame 1320 / 4062
I0306 14:05:24.284386    78 board_extractor.cc:443] Extracting corners from frame 1380 / 4062
I0306 14:05:26.340768    78 board_extractor.cc:443] Extracting corners from frame 1440 / 4062
I0306 14:05:28.352627    78 board_extractor.cc:443] Extracting corners from frame 1500 / 4062
I0306 14:05:30.375968    78 board_extractor.cc:443] Extracting corners from frame 1560 / 4062
I0306 14:05:32.285068    78 board_extractor.cc:443] Extracting corners from frame 1620 / 4062
I0306 14:05:34.274109    78 board_extractor.cc:443] Extracting corners from frame 1680 / 4062
I0306 14:05:36.254408    78 board_extractor.cc:443] Extracting corners from frame 1740 / 4062
I0306 14:05:38.407099    78 board_extractor.cc:443] Extracting corners from frame 1800 / 4062
I0306 14:05:40.439405    78 board_extractor.cc:443] Extracting corners from frame 1860 / 4062
I0306 14:05:42.557495    78 board_extractor.cc:443] Extracting corners from frame 1920 / 4062
I0306 14:05:44.681175    78 board_extractor.cc:443] Extracting corners from frame 1980 / 4062
I0306 14:05:46.752436    78 board_extractor.cc:443] Extracting corners from frame 2040 / 4062
I0306 14:05:48.889516    78 board_extractor.cc:443] Extracting corners from frame 2100 / 4062
I0306 14:05:50.976758    78 board_extractor.cc:443] Extracting corners from frame 2160 / 4062
I0306 14:05:53.016901    78 board_extractor.cc:443] Extracting corners from frame 2220 / 4062
I0306 14:05:55.096585    78 board_extractor.cc:443] Extracting corners from frame 2280 / 4062
I0306 14:05:57.129492    78 board_extractor.cc:443] Extracting corners from frame 2340 / 4062
I0306 14:05:59.254082    78 board_extractor.cc:443] Extracting corners from frame 2400 / 4062
I0306 14:06:01.556769    78 board_extractor.cc:443] Extracting corners from frame 2460 / 4062
I0306 14:06:03.716398    78 board_extractor.cc:443] Extracting corners from frame 2520 / 4062
I0306 14:06:05.945624    78 board_extractor.cc:443] Extracting corners from frame 2580 / 4062
I0306 14:06:07.992089    78 board_extractor.cc:443] Extracting corners from frame 2640 / 4062
I0306 14:06:09.947945    78 board_extractor.cc:443] Extracting corners from frame 2700 / 4062
I0306 14:06:12.069813    78 board_extractor.cc:443] Extracting corners from frame 2760 / 4062
I0306 14:06:14.257180    78 board_extractor.cc:443] Extracting corners from frame 2820 / 4062
I0306 14:06:16.562214    78 board_extractor.cc:443] Extracting corners from frame 2880 / 4062
I0306 14:06:18.791026    78 board_extractor.cc:443] Extracting corners from frame 2940 / 4062
I0306 14:06:21.005143    78 board_extractor.cc:443] Extracting corners from frame 3000 / 4062
I0306 14:06:23.411890    78 board_extractor.cc:443] Extracting corners from frame 3060 / 4062
I0306 14:06:25.585664    78 board_extractor.cc:443] Extracting corners from frame 3120 / 4062
I0306 14:06:27.735487    78 board_extractor.cc:443] Extracting corners from frame 3180 / 4062
I0306 14:06:29.849706    78 board_extractor.cc:443] Extracting corners from frame 3240 / 4062
I0306 14:06:31.837606    78 board_extractor.cc:443] Extracting corners from frame 3300 / 4062
I0306 14:06:33.993953    78 board_extractor.cc:443] Extracting corners from frame 3360 / 4062
I0306 14:06:36.165997    78 board_extractor.cc:443] Extracting corners from frame 3420 / 4062
I0306 14:06:38.217337    78 board_extractor.cc:443] Extracting corners from frame 3480 / 4062
I0306 14:06:40.261427    78 board_extractor.cc:443] Extracting corners from frame 3540 / 4062
I0306 14:06:42.368575    78 board_extractor.cc:443] Extracting corners from frame 3600 / 4062
I0306 14:06:44.481678    78 board_extractor.cc:443] Extracting corners from frame 3660 / 4062
I0306 14:06:46.619787    78 board_extractor.cc:443] Extracting corners from frame 3720 / 4062
I0306 14:06:48.876041    78 board_extractor.cc:443] Extracting corners from frame 3780 / 4062
I0306 14:06:50.957429    78 board_extractor.cc:443] Extracting corners from frame 3840 / 4062
I0306 14:06:53.089453    78 board_extractor.cc:443] Extracting corners from frame 3900 / 4062
I0306 14:06:55.203969    78 board_extractor.cc:443] Extracting corners from frame 3960 / 4062
I0306 14:06:57.288424    78 board_extractor.cc:443] Extracting corners from frame 4020 / 4062
Extracing corners for imu camera calibration.
I0306 14:06:59.176512   110 extract_board_to_json.cc:92] Starting board extraction. This might take a while...
Total number of frames: 4313
I0306 14:07:01.461062   110 board_extractor.cc:443] Extracting corners from frame 60 / 4313
I0306 14:07:03.474206   110 board_extractor.cc:443] Extracting corners from frame 120 / 4313
I0306 14:07:05.417107   110 board_extractor.cc:443] Extracting corners from frame 180 / 4313
I0306 14:07:07.567785   110 board_extractor.cc:443] Extracting corners from frame 240 / 4313
I0306 14:07:09.829749   110 board_extractor.cc:443] Extracting corners from frame 300 / 4313
I0306 14:07:11.922438   110 board_extractor.cc:443] Extracting corners from frame 360 / 4313
I0306 14:07:13.916007   110 board_extractor.cc:443] Extracting corners from frame 420 / 4313
I0306 14:07:15.887449   110 board_extractor.cc:443] Extracting corners from frame 480 / 4313
I0306 14:07:17.919632   110 board_extractor.cc:443] Extracting corners from frame 540 / 4313
I0306 14:07:19.929059   110 board_extractor.cc:443] Extracting corners from frame 600 / 4313
I0306 14:07:21.854396   110 board_extractor.cc:443] Extracting corners from frame 660 / 4313
I0306 14:07:23.835861   110 board_extractor.cc:443] Extracting corners from frame 720 / 4313
I0306 14:07:26.026546   110 board_extractor.cc:443] Extracting corners from frame 780 / 4313
I0306 14:07:28.180393   110 board_extractor.cc:443] Extracting corners from frame 840 / 4313
I0306 14:07:30.435138   110 board_extractor.cc:443] Extracting corners from frame 900 / 4313
I0306 14:07:32.521740   110 board_extractor.cc:443] Extracting corners from frame 960 / 4313
I0306 14:07:34.767199   110 board_extractor.cc:443] Extracting corners from frame 1020 / 4313
I0306 14:07:37.216979   110 board_extractor.cc:443] Extracting corners from frame 1080 / 4313
I0306 14:07:39.548312   110 board_extractor.cc:443] Extracting corners from frame 1140 / 4313
I0306 14:07:41.712136   110 board_extractor.cc:443] Extracting corners from frame 1200 / 4313
I0306 14:07:43.876083   110 board_extractor.cc:443] Extracting corners from frame 1260 / 4313
I0306 14:07:45.991308   110 board_extractor.cc:443] Extracting corners from frame 1320 / 4313
I0306 14:07:48.609725   110 board_extractor.cc:443] Extracting corners from frame 1380 / 4313
I0306 14:07:51.304021   110 board_extractor.cc:443] Extracting corners from frame 1440 / 4313
I0306 14:07:53.584003   110 board_extractor.cc:443] Extracting corners from frame 1500 / 4313
I0306 14:07:55.652161   110 board_extractor.cc:443] Extracting corners from frame 1560 / 4313
I0306 14:07:57.895082   110 board_extractor.cc:443] Extracting corners from frame 1620 / 4313
I0306 14:08:00.105944   110 board_extractor.cc:443] Extracting corners from frame 1680 / 4313
I0306 14:08:02.172910   110 board_extractor.cc:443] Extracting corners from frame 1740 / 4313
I0306 14:08:04.506282   110 board_extractor.cc:443] Extracting corners from frame 1800 / 4313
I0306 14:08:07.025419   110 board_extractor.cc:443] Extracting corners from frame 1860 / 4313
I0306 14:08:09.125464   110 board_extractor.cc:443] Extracting corners from frame 1920 / 4313
I0306 14:08:11.162721   110 board_extractor.cc:443] Extracting corners from frame 1980 / 4313
I0306 14:08:13.269928   110 board_extractor.cc:443] Extracting corners from frame 2040 / 4313
I0306 14:08:15.537608   110 board_extractor.cc:443] Extracting corners from frame 2100 / 4313
I0306 14:08:17.489622   110 board_extractor.cc:443] Extracting corners from frame 2160 / 4313
I0306 14:08:19.559062   110 board_extractor.cc:443] Extracting corners from frame 2220 / 4313
I0306 14:08:21.806123   110 board_extractor.cc:443] Extracting corners from frame 2280 / 4313
I0306 14:08:24.199276   110 board_extractor.cc:443] Extracting corners from frame 2340 / 4313
I0306 14:08:26.419615   110 board_extractor.cc:443] Extracting corners from frame 2400 / 4313
I0306 14:08:28.664405   110 board_extractor.cc:443] Extracting corners from frame 2460 / 4313
I0306 14:08:30.958622   110 board_extractor.cc:443] Extracting corners from frame 2520 / 4313
I0306 14:08:33.004849   110 board_extractor.cc:443] Extracting corners from frame 2580 / 4313
I0306 14:08:35.237669   110 board_extractor.cc:443] Extracting corners from frame 2640 / 4313
I0306 14:08:37.641304   110 board_extractor.cc:443] Extracting corners from frame 2700 / 4313
I0306 14:08:39.878690   110 board_extractor.cc:443] Extracting corners from frame 2760 / 4313
I0306 14:08:42.058080   110 board_extractor.cc:443] Extracting corners from frame 2820 / 4313
I0306 14:08:44.128227   110 board_extractor.cc:443] Extracting corners from frame 2880 / 4313
I0306 14:08:46.220291   110 board_extractor.cc:443] Extracting corners from frame 2940 / 4313
I0306 14:08:48.426939   110 board_extractor.cc:443] Extracting corners from frame 3000 / 4313
I0306 14:08:50.569886   110 board_extractor.cc:443] Extracting corners from frame 3060 / 4313
I0306 14:08:52.839723   110 board_extractor.cc:443] Extracting corners from frame 3120 / 4313
I0306 14:08:55.110695   110 board_extractor.cc:443] Extracting corners from frame 3180 / 4313
I0306 14:08:57.315148   110 board_extractor.cc:443] Extracting corners from frame 3240 / 4313
I0306 14:08:59.408208   110 board_extractor.cc:443] Extracting corners from frame 3300 / 4313
I0306 14:09:01.512667   110 board_extractor.cc:443] Extracting corners from frame 3360 / 4313
I0306 14:09:03.508982   110 board_extractor.cc:443] Extracting corners from frame 3420 / 4313
I0306 14:09:05.455116   110 board_extractor.cc:443] Extracting corners from frame 3480 / 4313
I0306 14:09:07.428071   110 board_extractor.cc:443] Extracting corners from frame 3540 / 4313
I0306 14:09:09.393474   110 board_extractor.cc:443] Extracting corners from frame 3600 / 4313
I0306 14:09:11.228258   110 board_extractor.cc:443] Extracting corners from frame 3660 / 4313
I0306 14:09:13.202042   110 board_extractor.cc:443] Extracting corners from frame 3720 / 4313
I0306 14:09:15.050997   110 board_extractor.cc:443] Extracting corners from frame 3780 / 4313
I0306 14:09:17.072211   110 board_extractor.cc:443] Extracting corners from frame 3840 / 4313
I0306 14:09:19.006461   110 board_extractor.cc:443] Extracting corners from frame 3900 / 4313
I0306 14:09:20.846570   110 board_extractor.cc:443] Extracting corners from frame 3960 / 4313
I0306 14:09:22.663596   110 board_extractor.cc:443] Extracting corners from frame 4020 / 4313
I0306 14:09:24.630484   110 board_extractor.cc:443] Extracting corners from frame 4080 / 4313
I0306 14:09:26.610970   110 board_extractor.cc:443] Extracting corners from frame 4140 / 4313
I0306 14:09:28.507961   110 board_extractor.cc:443] Extracting corners from frame 4200 / 4313
I0306 14:09:30.478192   110 board_extractor.cc:443] Extracting corners from frame 4260 / 4313
Finished corner extraction.
==================================================================
Corner extraction took 292.71s.
==================================================================
==================================================================
Running camera calibration.
==================================================================
Calibrating camera.
calibrate_camera: /OpenImuCameraCalibrator/include/OpenCameraCalibrator/utils/json.h:17346: const value_type& nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer>::operator[](T*) const [with T = const char; ObjectType = std::map; ArrayType = std::vector; StringType = std::__cxx11::basic_string<char>; BooleanType = bool; NumberIntegerType = long int; NumberUnsignedType = long unsigned int; NumberFloatType = double; AllocatorType = std::allocator; JSONSerializer = nlohmann::adl_serializer; nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer>::const_reference = const nlohmann::basic_json<>&; nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer>::value_type = nlohmann::basic_json<>]: Assertion `m_value.object->find(key) != m_value.object->end()' failed.
Finished camera calibration.
==================================================================
Camera calibration took 0.02s.
==================================================================
==================================================================
Extracting GoPro telemetry for imu bias and camera imu calibration.
==================================================================
Input video path: ../dataset/imu_bias/GX010029.MP4
Output file will be: ../dataset/imu_bias/GX010029.MP4_telemetry.json
Skipping GX010029 as telemetry file already exists: ../dataset/imu_bias/GX010029.json
Done
Input video path: ../dataset/cam_imu/GX010031.MP4
Output file will be: ../dataset/cam_imu/GX010031.MP4_telemetry.json
Opening ../dataset/cam_imu/GX010031.MP4
Read stream data from GX010031.MP4
Extracted information for GX010031.MP4
Flushing on end after 449137636 bytes
Generated telemetry for GX010031.MP4
Telemetry saved as JSON to GX010031.json
Done
==================================================================
Telemetry extraction took 1.37s.
==================================================================
==================================================================
Estimating IMU biases.
==================================================================
Estimated biases:
gyroscope bias:     0.00808 rad/s, -0.00500 rad/s, -0.00249 rad/s
accelerometer bias: 0.17300 m/s2,  -1.29628 m/s2,  0.00439 m/s2
Writing result to:  ../dataset/imu_bias/imu_bias_GX010029.json
==================================================================
IMU bias estimation took 0.97s.
==================================================================
==================================================================
Estimating camera poses for IMU - CAM calibration.
==================================================================
Could not open: ../dataset/cam/cam_calib_GX010027_di_2.json
F0306 14:09:35.376369   196 estimate_camera_poses_from_checkerboard.cc:54] Check failed: read_camera_calibration(FLAGS_camera_calibration_json, camera, fps) Could not read camera calibration: ../dataset/cam/cam_calib_GX010027_di_2.json
*** Check failure stack trace: ***
    @     0x7f3f53c821c3  google::LogMessage::Fail()
    @     0x7f3f53c8725b  google::LogMessage::SendToLog()
    @     0x7f3f53c81ebf  google::LogMessage::Flush()
    @     0x7f3f53c826ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x560a00020fb0  main
    @     0x7f3f51f07083  __libc_start_main
    @     0x560a0002096e  _start
==================================================================
Pose estimation estimation took 0.02s.
==================================================================
==================================================================
Estimating Spline error weighting and knot spacing.
==================================================================
Knot spacing SO3:               0.044 seconds at quality level q_so3=0.99
Knot spacing  R3:               0.150 seconds at quality level q_r3=0.99
Gyroscope weighting factor:     23.569 at quality level q_so3=0.99
Accelerometer weighting factor: 2.954 at quality level q_r3=0.99
Writing result to:  ../dataset/cam_imu/spline_info_GX010031.json
==================================================================
Spline weighting and knot spacing estimation took 1.03s.
==================================================================
==================================================================
Initializing IMU to camera rotation.
==================================================================
E0306 14:09:36.571174   229 reconstruction_reader.cc:65] Could not open the file: ../dataset/cam_imu/pose_calib_GX010031.calibdata for reading.
F0306 14:09:36.571293   229 estimate_imu_to_camera_rotation.cc:62] Check failed: theia::ReadReconstruction(FLAGS_input_pose_calibration_dataset, &pose_dataset) 
*** Check failure stack trace: ***
    @     0x7f12c9b981c3  google::LogMessage::Fail()
    @     0x7f12c9b9d25b  google::LogMessage::SendToLog()
    @     0x7f12c9b97ebf  google::LogMessage::Flush()
    @     0x7f12c9b986ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x556bdb151f16  main
    @     0x7f12c8649083  __libc_start_main
    @     0x556bdb1519be  _start
==================================================================
Spline weighting and knot spacing estimation took 0.17s.
==================================================================
==================================================================
Optimizing IMU to Camera calibration using Spline Fusion.
==================================================================
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0306 14:09:36.625880   230 reconstruction_reader.cc:65] Could not open the file: ../dataset/cam_imu/pose_calib_GX010031.calibdata for reading.
F0306 14:09:36.625944   230 continuous_time_imu_to_camera_calibration.cc:96] Check failed: theia::ReadReconstruction(FLAGS_input_pose_dataset, &pose_dataset) Could not read Reconstruction file.
*** Check failure stack trace: ***
==================================================================
Spline weighting and knot spacing estimation took 0.12s.
==================================================================
==================================================================
Print results.
==================================================================
Traceback (most recent call last):
  File "/OpenImuCameraCalibrator/python/../python/print_result_stats.py", line 98, in <module>
    main()
  File "/OpenImuCameraCalibrator/python/../python/print_result_stats.py", line 25, in main
    data = dict(data["trajectory"]) 
KeyError: 'trajectory'
root@7ce3ee00fab5:/OpenImuCameraCalibrator# 

Here's the latest run, with the latest built docker container Here's the dataset: https://drive.google.com/file/d/1zjuS25Fl3wZb4K6jGUXfCEeNNVtQbdGv/view?usp=sharing

[ajsnyde]

Btw when I see your underwater imagery, I am not sure if you can directly use the toolbox without modification. One probably needs to adjust some stuff according to that paper

I could be totally wrong here, but I interpreted the paper as looking to calibrate a camera or two above water and using some math to adapt the calibration for underwater usage. I'm looking to calibrate the camera underwater from the start (despite the dataset I'm uploading, that's to POC the IMU integration). Assuming you don't change mediums, the surrounding water is just another lens as far as SLAM is concerned.

Then again, I could be totally wrong and misinterpreting it, so if I am, let me know :smile:

[urbste]

I had a look at your sample dataset. The issue is, that you are using a standard checkerboard. This is not supported by my library. We need to make the orientation of the board w.r.t to the camera fix, to be able to align the coordinate systems of the board and the world (gravity) correctly. A checkerboard is not rotation invariant.

Please use a the Charuco board: https://github.com/urbste/OpenImuCameraCalibrator/blob/master/resource/board.png

[ajsnyde]

Thanks! I've redone the calibration videos using your board, and now it's hanging: Link to the new calibration dataset. Running command python3 python/run_gopro_calibration.py --path_calib_dataset ../dataset --checker_size_m=0.07436 --path_to_build ../OpenImuCameraCalibrator/build/applications/ --camera_model=FISHEYE https://drive.google.com/file/d/1WwSFUW4v7m_MphEzr3MWnLAJbQKUrnvU/view?usp=sharing

[ajsnyde]

Also trying downsized videos (like 30 mb instead of 500mb) to see if it's related to the resolution, but it's also hanging for 20+ minutes.

[urbste]

I think the reason is that there are too many images in the bundle adjustment. I also realized that the optimization is strangely slow in the docker container. Will have to debug it. In the meantime, just try to increase the --voxel_grid_size. This parameter is used to make sure that views from the video are not too close together for camera calibration. I just check if a camera poses is already within a cube of --voxel_grid_size (in meters) and if yes then the image is not added to the set of calibration images. Standard is 3cm in the script, so try increasing it to 5 or 6. You should not need more than 50-70 images for calibration.

[urbste]

I just --voxel_grid_size=0.06 and I get (with --image_downsample_factor=4):

Using 68 views for camera calibration. Final camera calibration reprojection error: 0.721298 from 47 view. Focal Length:425.1px Principal Point: 490.029/417.294px. FISHEYE model: Radial distortion 1: 0.0428467 Radial distortion 2: 0.212038 Radial distortion 3: -0.334151 Radial distortion 4: 0.0767752 Finished camera calibration.

[urbste]

What type of resolution setting is that? I have never seen 3840x3360?

[ajsnyde]

That's the latest GoPro Hero 11, with 4k 8:7 @60fps. No image stabilization (which tends to crop the video a bit).

[urbste]

Ah cool. Sure, for IMU-Camera calibration this needs to turned off anyways ;)

[urbste]

Ok now I am getting a weird error in the rotation initialization. I will check what's wrong there.

[urbste]

The camera poses for visual-inertial calibration look alright, maybe not enough motion

[ajsnyde]

I'll make a longer recording for the cam-imu data. Should I try to always keep the board taking up most of the video, or is it ok to back away from it a little?

[urbste]

It does not need to be in the entire view. Actually being a bit closer is better, The further away the more likely the detector will not find anymore corners. Actually I found the bug and it never appeared in my datasets :dancers: Just fixed it.

Calibration ran through, however I would only trust the rotation matrix. The translation vector is not correct. This is because a good translation estimate needs enough fast and diverse motion.

Mean reprojection error 4.74338 number residuals: 169129 I0308 20:36:07.658844 195327 continuous_time_imu_to_camera_calibration.cc:222] Mean reprojection error 4.74338px I0308 20:36:07.658856 195327 continuous_time_imu_to_camera_calibration.cc:223] Mean reprojection error after line delay optim 4.74338px g: 0.0167266 -9.87554 -1.46392 accel_bias at time 0: 0 0 0 gyro_bias at time 0: 0 0 0 T_i_c qw,qx,qy,qz: 0.00542101 -0.00092934 0.70593 -0.70826 T_i_c t: -0.0196559 -0.107959 -0.00654472 T_i_c R: -0.999939 0.00636687 0.00897014 -0.00899107 -0.00326617 -0.999954 -0.00633728 -0.999974 0.00332322 Initialized line delay [us]: 19.8611 Calibrated line delay [us]: 19.8611

[urbste]

Another tip: If you run the calibration multiple times, you can set --recompute_corners=0. Otherwise the corner extraction will be done every time and that takes forever ;)

[ajsnyde]

You freakin' rock man! Any way I can buy you a coffee or something?

[urbste]

Haha no problem. ;-) Actually, with that rotation estimate you can already run VI-SLAM. In my experience the IMU is very close to the projection center in GoPro cameras. Not having the translation vector correct will decrease the accuracy maybe a bit, but will not be a deal breaker.

[ajsnyde]

Eventually I'm going to move to a sliiiightly better IMU for ~300 bucks, and that will be placed ~30cm away, so I'll cross that road when it comes. I'm a matrix noob, how would I translate those numbers into ORBSLAM config? Looking at some existing IMU transformation, I see this: And noted that you gave me 16 numbers. Just want to make sure I have the matrix matched up to your numbers right.

[urbste]

Ok so Tbc means camera to body transformation (read from right to left). From my toolbox you get T_i_c which is the same direction camera -> body(imu). So you would set the ORB-SLAM3 matrix to

Tbc: !!opencv-matrix
   rows: 4
   cols: 4
   dt: f
   data: [ -0.999939 0.00636687 0.00897014,  0.0 , 
           -0.00899107 -0.00326617 -0.999954, 0.0,
          -0.00633728 -0.999974 0.00332322, 0.0,
          0.0, 0.0, 0.0, 1.0]

You see I set the translation vector (last column) to [0,0,0] because the estimate is currently not good enough.

By the looks of it, they did not change the IMU mounting much from previous GoPro generations. If you have a look at the values that I was using for my GoPro9 in ORB-SLAM3: config

[ajsnyde]

Nice to see. I'll give those values a shot on some footage I've got.

[ajsnyde]

Another rookie question, but how would one turn this:

Focal Length:841.945px Principal Point: 967.348/834.301px.
FISHEYE model: Radial distortion 1: 0.0485829 Radial distortion 2: 0.0209498 Radial distortion 3: 0.0145283 Radial distortion 4: -0.0188735
Finished camera calibration.

into an ORBSLAM yaml:

Would the camera fx and fy be the same focal length? Am I right to assume that the 1-4 radial distortion numbers are k1-k4? Am I using the right Camera.type for a fisheye action cam?

[urbste]

If you have an aspect ratio calibrated then fx!=fy: K = [fx 0 cx; 0 a*fy cy; 0 0 1]. Usually the aspect ratio is very small, as pixels should generally be square. Your values look alright though.

If I remember correctly, the fisheye model is the same and radial distortion 1-4 are equal to k1-k4.