Optimal intrinsic computation guideline

Tuhin-nd commented 5 months ago

Hi @lcit / @YinlinHu / @etrulls , I'm using your repo for calibration but during Intrinsics computation with uniform keypoint detection distribution the reprojection error is coming very high ~ 3.88

Montonicity: monotonicity

Intrinsics: intrinsics.json

Detected keypoints: detected_keypoints

Please suggest what should be the recipe to get best intrinsics result.

lcit commented 5 months ago

How much error do you get if you run the example in examples/intrinsics? And, what values have you used for --inner_corners_height and -inner_corners_width ?

Tuhin-nd commented 5 months ago

@lcit thanks for replying.

inner_corners_height = 9 inner_corners_width = 14

Here are the results after I ran this script "examples/intrinsics/example.py" with one sample frame from one of my cameras (same as to what is shown in the previous thread) using the above height and width parameters : sample_example_output

As we can see from the image there is some error on the corners. Let me know your views on this.

lcit commented 5 months ago

The inner corners parameters are correct. Regarding the example, can you run the example as is on the images that are already in the folder? That means executing bash run_extract_frames.sh; bash run_intrinsics_estimation.sh. This is to make sure that the library and opencv version are working. If that gives a low reprojection error then I suggest retaking the images in the same way as in this example vid_pattern.mp4 while paying attentions to these:

Auto-focus, auto-rotation and focal length must be disabled or fixed.
The squares of the checkerboard must be square, sharp and flat.
The images must be sharp. If you take a video move slowly.
50-100 image is usually enough
The images should be taken such that the checkerboard covers every part of the image and are evenly spread.
During the capture, you should rotate the camera left-right and up-down and repeat this operation at different positions.

Tuhin-nd commented 5 months ago

I ran the experiment using vid_pattern.mp4 Output:

tuhindutta@nd-blr-tuhind intrinsics % bash run_extract_frames.sh 
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.3.9.4)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.1.1_6 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopenvino --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'vid_pattern.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:01:12.00, start: 0.000000, bitrate: 681 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 800x500 [SAR 10:9 DAR 16:9], 681 kb/s, 1 fps, 1 tbr, 16384 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> png (native))
Press [q] to stop, [?] for help
[swscaler @ 0x1202f8000] [swscaler @ 0x130158000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120308000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120318000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120328000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120338000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120348000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x120358000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x128518000] No accelerated colorspace conversion found from yuv420p to rgb24.
[swscaler @ 0x1202f8000] [swscaler @ 0x128528000] No accelerated colorspace conversion found from yuv420p to rgb24.
Output #0, image2, to 'frames/frame_%04d.png':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.16.100
  Stream #0:0(und): Video: png, rgb24(pc, gbr/unknown/unknown, progressive), 800x500 [SAR 10:9 DAR 16:9], q=2-31, 200 kb/s, 1 fps, 1 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 png
[out#0/image2 @ 0x13a627cb0] video:39217kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=   72 fps=0.0 q=-0.0 Lsize=N/A time=00:01:11.00 bitrate=N/A speed= 128x    
tuhindutta@nd-blr-tuhind intrinsics % conda activate camera-calib
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % which python
/Users/tuhindutta/anaconda3/envs/camera-calib/bin/python
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % which python3
/Users/tuhindutta/anaconda3/envs/camera-calib/bin/python3
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % python3 --version
Python 3.9.19
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % bash run_intrinsics_estimation.sh
python: can't open file '/Users/tuhindutta/Workspaces/ai360-calibration/scripts/compute_intrinsics.py': [Errno 2] No such file or directory
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % bash run_intrinsics_estimation.sh
Traceback (most recent call last):
  File "/Users/tuhindutta/Workspaces/ai360-calibration/scripts/compute_intrinsics.py", line 35, in <module>
    from multiview_calib import utils
ModuleNotFoundError: No module named 'multiview_calib'
(camera-calib) tuhindutta@nd-blr-tuhind intrinsics % cd .. 
(camera-calib) tuhindutta@nd-blr-tuhind examples % cd ..
(camera-calib) tuhindutta@nd-blr-tuhind ai360-calibration % python scripts/compute_intrinsics.py --folder_images examples/intrinsics/frames/ -ich 6 -icw 8 -s 30 -t 24 --debug -rm
Traceback (most recent call last):
  File "/Users/tuhindutta/Workspaces/ai360-calibration/scripts/compute_intrinsics.py", line 35, in <module>
    from multiview_calib import utils
ModuleNotFoundError: No module named 'multiview_calib'
(camera-calib) tuhindutta@nd-blr-tuhind ai360-calibration % python compute_intrinsics.py --folder_images examples/intrinsics/frames/ -ich 6 -icw 8 -s 30 -t 24 --debug -rm 
--------------------------------------------------
Input parameters
--------------------------------------------------
folder_images: examples/intrinsics/frames/
output_folder: ./output
description: 
inner_corners_height: 6
inner_corners_width: 8
square_sizes: 30
rational_model: True
alpha: 0.95
force_monotonicity: False
monotonic_range: -1
intrinsic_guess: False
fix_principal_point: False
fix_aspect_ratio: False
zero_tangent_dist: False
criteria_eps: 1e-05
threads: 24
debug: True
--------------------------------------------------
Processing image examples/intrinsics/frames/frame_0001.png ...
Processing image examples/intrinsics/frames/frame_0002.png ...
Processing image examples/intrinsics/frames/frame_0003.png ...
Processing image examples/intrinsics/frames/frame_0004.png ...
Processing image examples/intrinsics/frames/frame_0005.png ...
Processing image examples/intrinsics/frames/frame_0006.png ...
Processing image examples/intrinsics/frames/frame_0007.png ...
Processing image examples/intrinsics/frames/frame_0008.png ...
Processing image examples/intrinsics/frames/frame_0009.png ...
Processing image examples/intrinsics/frames/frame_0010.png ...
Processing image examples/intrinsics/frames/frame_0011.png ...
Processing image examples/intrinsics/frames/frame_0012.png ...
Processing image examples/intrinsics/frames/frame_0013.png ...
Processing image examples/intrinsics/frames/frame_0014.png ...
Processing image examples/intrinsics/frames/frame_0015.png ...
Processing image examples/intrinsics/frames/frame_0016.png ...
Processing image examples/intrinsics/frames/frame_0017.png ...
Processing image examples/intrinsics/frames/frame_0018.png ...
Processing image examples/intrinsics/frames/frame_0019.png ...
Processing image examples/intrinsics/frames/frame_0020.png ...
Processing image examples/intrinsics/frames/frame_0021.png ...
Processing image examples/intrinsics/frames/frame_0022.png ...
Processing image examples/intrinsics/frames/frame_0023.png ...
Processing image examples/intrinsics/frames/frame_0024.png ...
Processing image examples/intrinsics/frames/frame_0025.png ...
Processing image examples/intrinsics/frames/frame_0026.png ...
Processing image examples/intrinsics/frames/frame_0027.png ...
Processing image examples/intrinsics/frames/frame_0028.png ...
Processing image examples/intrinsics/frames/frame_0029.png ...
Processing image examples/intrinsics/frames/frame_0030.png ...
Processing image examples/intrinsics/frames/frame_0031.png ...
Processing image examples/intrinsics/frames/frame_0032.png ...
Processing image examples/intrinsics/frames/frame_0033.png ...
Processing image examples/intrinsics/frames/frame_0034.png ...
Processing image examples/intrinsics/frames/frame_0035.png ...
Processing image examples/intrinsics/frames/frame_0036.png ...
Processing image examples/intrinsics/frames/frame_0037.png ...
Processing image examples/intrinsics/frames/frame_0038.png ...
Processing image examples/intrinsics/frames/frame_0039.png ...
Processing image examples/intrinsics/frames/frame_0040.png ...
Processing image examples/intrinsics/frames/frame_0041.png ...
Processing image examples/intrinsics/frames/frame_0042.png ...
Processing image examples/intrinsics/frames/frame_0043.png ...
Processing image examples/intrinsics/frames/frame_0044.png ...
Processing image examples/intrinsics/frames/frame_0045.png ...
Processing image examples/intrinsics/frames/frame_0046.png ...
Processing image examples/intrinsics/frames/frame_0047.png ...
Processing image examples/intrinsics/frames/frame_0048.png ...
Processing image examples/intrinsics/frames/frame_0049.png ...
Processing image examples/intrinsics/frames/frame_0050.png ...
Processing image examples/intrinsics/frames/frame_0051.png ...
Processing image examples/intrinsics/frames/frame_0052.png ...
Processing image examples/intrinsics/frames/frame_0053.png ...
Processing image examples/intrinsics/frames/frame_0054.png ...
Processing image examples/intrinsics/frames/frame_0055.png ...
Processing image examples/intrinsics/frames/frame_0056.png ...
Processing image examples/intrinsics/frames/frame_0057.png ...
Processing image examples/intrinsics/frames/frame_0058.png ...
Processing image examples/intrinsics/frames/frame_0059.png ...
Processing image examples/intrinsics/frames/frame_0060.png ...
Processing image examples/intrinsics/frames/frame_0061.png ...
Processing image examples/intrinsics/frames/frame_0062.png ...
Processing image examples/intrinsics/frames/frame_0063.png ...
Processing image examples/intrinsics/frames/frame_0064.png ...
Processing image examples/intrinsics/frames/frame_0065.png ...
Processing image examples/intrinsics/frames/frame_0066.png ...
Processing image examples/intrinsics/frames/frame_0067.png ...
Processing image examples/intrinsics/frames/frame_0068.png ...
Processing image examples/intrinsics/frames/frame_0069.png ...
Processing image examples/intrinsics/frames/frame_0070.png ...
Processing image examples/intrinsics/frames/frame_0071.png ...
Processing image examples/intrinsics/frames/frame_0072.png ...
72
71
working hard...
Reprojection error for frame 0: 0.06140914514879008
Reprojection error for frame 1: 0.06118428174609205
Reprojection error for frame 2: 0.06035583788975876
Reprojection error for frame 3: 0.06021101847769241
Reprojection error for frame 4: 0.056587353228253284
Reprojection error for frame 5: 0.0556997718352619
Reprojection error for frame 6: 0.058797614661658976
Reprojection error for frame 7: 0.0583646758671282
Reprojection error for frame 8: 0.059176789091151094
Reprojection error for frame 9: 0.05974726168380922
Reprojection error for frame 10: 0.06052313543748478
Reprojection error for frame 11: 0.06564291444389142
Reprojection error for frame 12: 0.06207452084926335
Reprojection error for frame 13: 0.05727683880941134
Reprojection error for frame 14: 0.05824928503502974
Reprojection error for frame 15: 0.0466806950780134
Reprojection error for frame 16: 0.06091809004439311
Reprojection error for frame 17: 0.06653084479472507
Reprojection error for frame 18: 0.06557544848065744
Reprojection error for frame 19: 0.06793125718791045
Reprojection error for frame 20: 0.06570019697009404
Reprojection error for frame 21: 0.04768028859855623
Reprojection error for frame 22: 0.06539256566758256
Reprojection error for frame 23: 0.0458266680619883
Reprojection error for frame 24: 0.06445819099502408
Reprojection error for frame 25: 0.06653767033651718
Reprojection error for frame 26: 0.07007157389210315
Reprojection error for frame 27: 0.0724851056801564
Reprojection error for frame 28: 0.06337499016346253
Reprojection error for frame 29: 0.06113641445433882
Reprojection error for frame 30: 0.058723216066821786
Reprojection error for frame 31: 0.04381420128053586
Reprojection error for frame 32: 0.04318384373163056
Reprojection error for frame 33: 0.04303741643635675
Reprojection error for frame 34: 0.04324404593581415
Reprojection error for frame 35: 0.04309007246922671
Reprojection error for frame 36: 0.04245927294042721
Reprojection error for frame 37: 0.04163030214144873
Reprojection error for frame 38: 0.04048481424165223
Reprojection error for frame 39: 0.041862308202556486
Reprojection error for frame 40: 0.0442263687298726
Reprojection error for frame 41: 0.04819399401859773
Reprojection error for frame 42: 0.04921976953416959
Reprojection error for frame 43: 0.04967111273096886
Reprojection error for frame 44: 0.06643924018644225
Reprojection error for frame 45: 0.06615244702230734
Reprojection error for frame 46: 0.04452016949285548
Reprojection error for frame 47: 0.043203362330643445
Reprojection error for frame 48: 0.04184510894345095
Reprojection error for frame 49: 0.042797801538742776
Reprojection error for frame 50: 0.05852459369221392
Reprojection error for frame 51: 0.06393344931214869
Reprojection error for frame 52: 0.0666651187963082
Reprojection error for frame 53: 0.06847198715755681
Reprojection error for frame 54: 0.07165000837356622
Reprojection error for frame 55: 0.0767073672099975
Reprojection error for frame 56: 0.07126333785400743
Reprojection error for frame 57: 0.0615596878109727
Reprojection error for frame 58: 0.06638440982117286
Reprojection error for frame 59: 0.06830772424688865
Reprojection error for frame 60: 0.06558711986804103
Reprojection error for frame 61: 0.06580188182639576
Reprojection error for frame 62: 0.06356156223119945
Reprojection error for frame 63: 0.06372186498947352
Reprojection error for frame 64: 0.06336550041063468
Reprojection error for frame 65: 0.05798687933141263
Reprojection error for frame 66: 0.06093716758373204
Reprojection error for frame 67: 0.0693931941543209
Reprojection error for frame 68: 0.0689353648979918
Reprojection error for frame 69: 0.06084861756430659
Reprojection error for frame 70: 0.06576375436541694
RMS Reprojection Error: 0.1277961014540219, Total Reprojection Error: 0.058348871888485636
Saving undistorted images..

intrinsics.json:

intrinsics.json

monotonicity:

detected_keypoints:

Uploading few undistorted images:

frame_0023

frame_0002

frame_0069

frame_0050

Tuhin-nd commented 5 months ago

Please help me debug if the below factors have any impact on the calibration -

Does the below two factors affect the calibration re-projection error -
- bigger checkerboard pattern,
- checkerboards being a bit old have got de-colored from pure black to faded reddish ?
Also what if I use a A4 size paper checkerboard instead of the bigger one I'm currently using, will it affect the results?
How much does flickering lighting due to bitrate affect the calibration output as I'm streaming the video from the camera device to my system?

lcit commented 5 months ago

So the example works. A think that the most likely cause of the failure is the way you take the images. Do you move the camera or you keep it always in the same position? Try to add more variance in rotation and position.

Regarding the questions:

I do not think so, If the detections are precise it does not matter
You can try but it should not matter
Probably none. What matter is the detection accuracy and that the lens/focal do not change.

Tuhin-nd commented 4 months ago

Thank you for your valuable insights w.r.t. my questions.

As you mentioned, I found by experimenting is that recording the video the "perfect" way (the guidelines you mentioned) has the utmost impact in getting the parameters and undistorted images right. Only thing I could not follow is the sharpness of the images as there are camera limitations.

The results have improved after using smaller checkerboard 6x8 with 30mm square size as it is lot easier to cover the whole image space and rotate the camera left-right and up-down ways when we use smaller A4 size checkerboard.

The reprojection errors are now < 1 for all cameras (including wider and flat lenses).

cvlab-epfl / multiview_calib

Optimal intrinsic computation guideline #13