Car Surround view with fisheye cameras

[mrp1234567]

Hi,

I need to use libxcam to stitch the frames from 4 cameras around the car.

Can someone help me with detailed steps i need to do ? tools to be used, required changes to any of the source files of libxcam etc

https://sites.google.com/site/scarabotix/ocamcalib-omnidirectional-camera-calibration-toolbox-for-matlab does this tool generate extrinsic parameters ?

I built libxcam and used sample images provided in ocamcalib zip file for calibration and it generated calib_results.txt and Omni_Calib_Results.mat files

[zongwave]

Hi, the 4 cameras surround view stitching using Scaramuzza's oCamCalib tool to do camera calibration, you should generate extrinsic & intrinsic parameters, saved parameters into text files as: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L35 https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L42

libXCam implemented calibration parser at: https://github.com/intel/libxcam/blob/master/xcore/calibration_parser.h#L38

[mrp1234567]

@zongwave Thank You for the response.

I generated the calibration file from oCamCalib tool which has intrinsic parameters(calib_results.txt), how do I generate extrinsic parameters ?

In their documentation they mentioned RRFin has extrinsic parameters for each calibrated frame, not sure how to use RRFin to get extrinsic parameters in the following format, appreciate if you can help on this ?

extrinsic fisheye camera calibration, positions in millimeters, rotations in degrees

Position x

x

Position y

y

Position z

z

Rotation x (roll)

r

Rotation x (pitch)

p

Rotation x (yaw)

yy

[zongwave]

There's a discussion about RRFin, you can check it out. https://github.com/intel/libxcam/issues/616

[mrp1234567]

@zongwave

I have the extrinsic parameters (rotations in degrees) and intrinsics from oCamCalib but the output of four input images(in nv12 format) is still not good

I am using the following command, 4 cameras are fisheye (stereographic lens)

./test-surround-view --module soft --input frame_0_0_0.nv12 --input frame_0_0_1.nv12 --input frame_0_0_2.nv12 --input frame_0_0_3.nv12 --output test-surround-view_output.nv12 --in-w 1280 --in-h 960 --in-format nv12 --fisheye-num 4 --save true --save-topview true --loop 1

What else to look for to debug ?

What difference does --cam-model make ? [cama2c1080p/camb4c1080p/camc3c8k/camc6c8k/camd3c8k], default: camb4c1080p

[zongwave]

cama2c1080p is used to identify 2 cameras with 1080p and parse the stitching parameters, camb4c1080p is for 4 cameras with 1080p, camc3c8k is for 3 cameras with 8k stitched output, camc6c8k is for 6 cameras with 8k stitch output.

We tested stitching function with video clips recorded by these cameras', you can find these parameters at: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368

[zongwave]

please enable stitching intermedia image dumping: https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34 https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47

and if you enable opecv feature match, also enable here: https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

[mrp1234567]

camb4c1080p

1. Our cameras output 1280x960 resolution frames, is it still ok to use camb4c1080p model ?

2. I do not see camb4c1080p in https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368, am I supposed to make any code changes other than giving input calibration parameters and 4 images from 4 cameras to generate surround view ?

3. You mentioned you used video clips but as start up test, I am giving just 4 images to stitch, is that fine or videos are mandatory ?

[mrp1234567]

please enable stitching intermedia image dumping: https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34 https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47

and if you enable opecv feature match, also enable here: https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

1. Can I know the list of all configurations that are supposed to be done ?
2. Is there a need to configure vehicle measurements information? If so, where do i need to configure ?
3. is ZYX convention for Euler Angles correct ?

Can you please verify the "Steps Followed" are correct ? Steps Followed: 1. Used oCamCalib to generate intrinsic parameters(calib_results.txt) for all the 4 cameras 2. From RRfin, cross product of column 1 and column2 to generate column 3, then rotation matrix is converted to Euler Angles(ZYX) 3. Eluer Angles(in degrees)(roll - x, pitcy - y, yaw - z) are used in corresponding extrinsic files 4. Last column from RRfin of oCamCalib tool are used as Position x, Position Y and Position Z in corresponding extrinsic files 5. RRfin of an image where checkerboard is lying flat on the ground is taken for extrinsic parameter for each camera 6. Following is RRfin for one camera and calculations and Euler Angles val(:,:,11) =

1.0e+02 *

-0.000093441273664 0.004960374800235 0.246741453624884 -0.009943325487015 0.000873493262281 0.364232264603431 0.001059031155184 0.008638963558318 -2.959915923124880

Euler Angles(ZYX) from rotation matrix: radians: [ x: 1.0527373, y: -0.1061021, z: -1.5801934 ] degrees: [ x: 60.3174068, y: -6.0792012, z: -90.5384151 ]

MatLab: LeftRRFinColumn1 = -0.009344127366400 -0.994332548701500 0.105903115518400 LeftRRFinColumn2 = 0.496037480023500 0.087349326228100 0.863896355831800 LeftRRFinColumn3 = cross(LeftRRFinColumn1, LeftRRFinColumn2)

LeftRRFinColumn3 = -0.868250831094161 0.060604272128646 0.492410008533592

7. Content of extrinsic_camera_left.txt based on above RRfin

Position x

24.6741453624884

Position y

36.4232264603431

Position z

-295.9915923124880

Rotation x (roll)

1.0527373

60.3174068

Rotation x (pitch)

-0.1061021

-6.0792012

Rotation x (yaw)

-1.5801934

-90.538415189

8. Following files are placed in FISHEYE_CONFIG_PATH extrinsic_camera_front.txt extrinsic_camera_rear.txt intrinsic_camera_front.txt intrinsic_camera_rear.txt extrinsic_camera_left.txt extrinsic_camera_right.txt intrinsic_camera_left.txt intrinsic_camera_right.txt

[mrp1234567]

stitcher-debug files.zip test-surround-view_output.zip

Attached the debug files generated with XCAM_TEST_STREAM_DEBUG and DUMP_STITCHER enabled and also the generated output file

Can you please let me know, what might be going wrong that output is not good.

[zongwave]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.

BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ?

[zongwave]

camb4c1080p

1. Our cameras output 1280x960 resolution frames, is it still ok to use camb4c1080p model ?
2. I do not see camb4c1080p in https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368, am I supposed to make any code changes other than giving input calibration parameters and 4 images from 4 cameras to generate surround view ?
3. You mentioned you used video clips but as start up test, I am giving just 4 images to stitch, is that fine or videos are mandatory ?

this enum camb4c1080p is to define bowl model config https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L128, and each camera's fov https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L154 and feature match configuration: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L240

You should config these parameters according to your own camera hardware configuration.

You can apply still images in your development at the beginning.

[zongwave]

please enable stitching intermedia image dumping: https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34 https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47 and if you enable opecv feature match, also enable here: https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

1. Can I know the list of all configurations that are supposed to be done ?
2. Is there a need to configure vehicle measurements information? If so, where do i need to configure ?
3. is ZYX convention for Euler Angles correct ?

Can you please verify the "Steps Followed" are correct ? Steps Followed: 1. Used oCamCalib to generate intrinsic parameters(calib_results.txt) for all the 4 cameras 2. From RRfin, cross product of column 1 and column2 to generate column 3, then rotation matrix is converted to Euler Angles(ZYX) 3. Eluer Angles(in degrees)(roll - x, pitcy - y, yaw - z) are used in corresponding extrinsic files 4. Last column from RRfin of oCamCalib tool are used as Position x, Position Y and Position Z in corresponding extrinsic files 5. RRfin of an image where checkerboard is lying flat on the ground is taken for extrinsic parameter for each camera 6. Following is RRfin for one camera and calculations and Euler Angles val(:,:,11) =

1.0e+02 *

-0.000093441273664 0.004960374800235 0.246741453624884 -0.009943325487015 0.000873493262281 0.364232264603431 0.001059031155184 0.008638963558318 -2.959915923124880

Euler Angles(ZYX) from rotation matrix: radians: [ x: 1.0527373, y: -0.1061021, z: -1.5801934 ] degrees: [ x: 60.3174068, y: -6.0792012, z: -90.5384151 ]

MatLab: LeftRRFinColumn1 = -0.009344127366400 -0.994332548701500 0.105903115518400 LeftRRFinColumn2 = 0.496037480023500 0.087349326228100 0.863896355831800 LeftRRFinColumn3 = cross(LeftRRFinColumn1, LeftRRFinColumn2)

LeftRRFinColumn3 = -0.868250831094161 0.060604272128646 0.492410008533592

7. Content of extrinsic_camera_left.txt based on above RRfin #Position x 24.6741453624884 #Position y 36.4232264603431 #Position z -295.9915923124880 #Rotation x (roll) #1.0527373 60.3174068 #Rotation x (pitch) #-0.1061021 -6.0792012 #Rotation x (yaw) #-1.5801934 -90.538415189

8. Following files are placed in FISHEYE_CONFIG_PATH extrinsic_camera_front.txt extrinsic_camera_rear.txt intrinsic_camera_front.txt intrinsic_camera_rear.txt extrinsic_camera_left.txt extrinsic_camera_right.txt intrinsic_camera_left.txt intrinsic_camera_right.txt

you need import your cameras intrinsic & extrinsic parameters, and map the four camera images into a bowl model, https://github.com/intel/libxcam/blob/master/xcore/interface/data_types.h#L188 The extrinsic parameters represents your vehicle measurements information in some extent.

[mrp1234567]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.

BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ? orig_fisheye_and_Output_files.zip

Attached the orig files and output files

[zongwave]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images. BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ? orig_fisheye_and_Output_files.zip

Attached the orig files and output files

did you set your input & output frame size correctly? you input four 1280x960 images, after stitching output same resolution images. basically, the output image's width should be larger than height, because after stitching we got 360 degree fov in horizontal direction and less than 180 degree fov in vertical direction. here's the default input & output resolution settings: https://github.com/intel/libxcam/blob/master/tests/test-surround-view.cpp#L560

[mrp1234567]

Yes, I tried both 1280x960 and also default 1920x640 for output resolution But I think I need to change following files as you mentioned, right now I am using whatever is there in github directly https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L128, and each camera's fov https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L154 and feature match configuration: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L240 https://github.com/intel/libxcam/blob/master/xcore/interface/data_types.h#L188

[mrp1234567]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.

BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ?

stitcher-geomap-xx.NV12.yuv is not good means calibration is not good ?

[mrp1234567]

camb4c1080p

1. Our cameras output 1280x960 resolution frames, is it still ok to use camb4c1080p model ?
2. I do not see camb4c1080p in https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368, am I supposed to make any code changes other than giving input calibration parameters and 4 images from 4 cameras to generate surround view ?
3. You mentioned you used video clips but as start up test, I am giving just 4 images to stitch, is that fine or videos are mandatory ?

this enum camb4c1080p is to define bowl model config https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L128, and each camera's fov https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L154 and feature match configuration: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L240

You should config these parameters according to your own camera hardware configuration.

You can apply still images in your development at the beginning.

Attached the toy car setup i have with 4 fisheye cameras on four directions.

1. Each camera is around 200mm from the ground, Can I know What I am supposed to set for a,b,c,center_z, wall_height, ground_length ? Based on what those values should be derived ?

2. Also can you please let me know, how to select the parameters for soft_fm_config() ?

3. When I set all the four cameras range as 180 degrees, I am getting following assertion

Can you please help with the configuration parameters ? Is there any user guide for configuration parameters ?

[zongwave]

You should config the "viewpoints_range" parameters according to your camera's FoV this diagram illustrates how the camera layout and FoV

You also should define the output stitch frame size, then the algorithm calculate the correction frame (out from geomap) size and estimate the overlap width between each stitching pairs.

these pixels out of overlap areas, we will start some tasks simply copy these pixels into final frame buffer. I think the assert you mentioned above is due to copy pixel estimation is ZERO!

[mrp1234567]

Hi Wei Zong,

1. Irrespective of what value I use for output width(1280/1920/2048/3076/4096), always there is assertion when I use 180 for all the four cameras in viewpoints_range() for CamB4C1080P: { range[0] = 180.0f; range[1] = 180.0f; range[2] = 180.0f; range[3] = 180.0f; break; } How much output width am I supposed to put and based on what the output width should be set ?

2. Can I know What I am supposed to set for a,b,c,center_z, wall_height, ground_length in bowl_config() ? Based on what, those values should be set ?

3. How to select FMConfig parameters for CamB4C1080P in soft_fm_config()?

4. What is the reference point/base link for extrinsic parameters ?

5. For fisheye cameras, do we need to feed separately json file with intrinsic and extrinsic parameters ?

6. When to use --dewarp-mode=sphere vs bowl ?

7. Do you have a Software Detailed Design document and User guide for configuration parameters ?

Can you please answer all the line items specifically under each question?

Rgds Mahipal

[zongwave]

Hi Wei Zong,

1. Irrespective of what value I use for output width(1280/1920/2048/3076/4096), always there is assertion when I use 180 for all the four cameras in viewpoints_range() for CamB4C1080P: { range[0] = 180.0f; range[1] = 180.0f; range[2] = 180.0f; range[3] = 180.0f; break; } How much output width am I supposed to put and based on what the output width should be set ?
2. Can I know What I am supposed to set for a,b,c,center_z, wall_height, ground_length in bowl_config() ? Based on what, those values should be set ?
3. How to select FMConfig parameters for CamB4C1080P in soft_fm_config()?
4. What is the reference point/base link for extrinsic parameters ?
5. For fisheye cameras, do we need to feed separately json file with intrinsic and extrinsic parameters ?
6. When to use --dewarp-mode=sphere vs bowl ?
7. Do you have a Software Detailed Design document and User guide for configuration parameters ?

Can you please answer all the line items specifically under each question?

Rgds Mahipal

1. the "viewpoints_range" represents your camera's field of view. please read the diagram in my last comments. if you set 180 degrees for all your four cameras, that means all pixels in your input frames belong to "overlap" areas which need to be stitching and blending together, that will be unnecessary.

2. this bowl_config is used to build a bowl view model, it used to rendering a stitched frame into a bowl view. that define how your bowl view looks like, i.e. the aspect ratio of the long and short axis of ellipsoidal. it is not related to stitching process, you can ignore this bowl modeling in stitching.

3. FMConfig parameters are used to do key points feature match in stitching. libXCam use OpenCV optical flow algorithm to do feature match between two stitching images. you can check the code by yourself for details.

4. In previous discussion I posted Scaramuzza's website that described the camera calibration tools very detailed. I post this website for you again, I wish you can read the contents carefully. https://sites.google.com/site/scarabotix/ocamcalib-omnidirectional-camera-calibration-toolbox-for-matlab

5. you can prepare a json file to restore your camera calibration parameters, so that you will no need to change parameters in source code and build the source code.

6. dewarp mode "sphere" is used to generate a 360 degree video. the modeling is a Sphere. We have a diagram in libXCam's Wiki https://github.com/intel/libxcam/wiki#libxcam-stitch-processing-flow

7. libXCam Wiki record some information about software framework, supported features, build instruction, and so on, maybe I can prepare some document later.

Someone opened another issue want to see how this surround view feature work, I just use the input images you posted in previous comments to process a surround view stitching. I do NOT get a properly camera calibration files from you, so the fisheye correction is not accurate. the overall quality is not satisfied.

A processing demo from the images you provided. I do NOT get a properly camera calibration files from you

[mrp1234567]

Hi Wei Zong,

Thank you so much for the detailed answers.

1. I am confused with your answer for viewpoints_range, "viewpoints_range" represents your camera's field of view and if you set 180 degrees for all your four cameras, that means all pixels in your input frames belong to "overlap" areas which need to be stitching and blending together, that will be unnecessary. These two statements are confusing If all my cameras have 180 fov, what to set for viewpoints_range(), what's the algorithm that is supposed to be used to set the viewpoints_range for different fov scenarios ?

2. Regarding the reading of ocamcalib documentation, I am not sure which point you are refring to, with respect to extrinsic parameters. I see this statement "Contains the extrinsic parameters of every checkerboard with the respect to the camera reference frame.", but this is for each camera, how libxcam expects the extrinsic parameters ? Which reference does libxcam takes ? Does it transform given extrinsics to some other reference point before they are being used by the algorithm ?

3. With OCamCalib generated intrinsics and extrinsics, I am getting the following output, which is very bad, where as the one which you generated is far better. Can you please let me know what all configurations you used and also if you did any changes to calibration parameters ?

Rgds Mahipal

[zongwave]

Hi Wei Zong,

Thank you so much for the detailed answers.

1. I am confused with your answer for viewpoints_range, "viewpoints_range" represents your camera's field of view and if you set 180 degrees for all your four cameras, that means all pixels in your input frames belong to "overlap" areas which need to be stitching and blending together, that will be unnecessary. These two statements are confusing If all my cameras have 180 fov, what to set for viewpoints_range(), what's the algorithm that is supposed to be used to set the viewpoints_range for different fov scenarios ?
2. Regarding the reading of ocamcalib documentation, I am not sure which point you are refring to, with respect to extrinsic parameters. I see this statement "Contains the extrinsic parameters of every checkerboard with the respect to the camera reference frame.", but this is for each camera, how libxcam expects the extrinsic parameters ? Which reference does libxcam takes ? Does it transform given extrinsics to some other reference point before they are being used by the algorithm ?
3. With OCamCalib generated intrinsics and extrinsics, I am getting the following output, which is very bad, where as the one which you generated is far better. Can you please let me know what all configurations you used and also if you did any changes to calibration parameters ?

Rgds Mahipal

if you use 4 FoV 180 degree cameras (installed in front, left, rear, right four directions) for 360 degree capturing. then the overlap area between each cameras will be too large. the pixels involved in stitching & blending will be twice as a 360 view pixels. (4 x 180 = 2 x 360). this is unnecessary, it wastes computing and memory. and in worse case will bring heavy ghost effect. In our implementation, we use cameras with reasonable FoV and set properly "viewpoints_range" value accordingly. Those pixels are not falling in overlap areas will skip stitching & blending, just copy these pixels into final stitch frame buffer.

You can compare this diagram with another diagram I posted before to understand the process.

[mengyafei43]

@mrp1234567 Did you run it success now, using your [orig_fisheye_and_Output_files.zip], can you share your "extrinsic_camera_front.txt extrinsic_camera_rear.txt intrinsic_camera_front.txt intrinsic_camera_rear.txt extrinsic_camera_left.txt extrinsic_camera_right.txt intrinsic_camera_left.txt intrinsic_camera_right.txt". correspond with you orig_fisheye debug images.