Question regarding this example about the OCamCalib model

[Sheradil]

Hey,

now that I fixed my previous issue, I did some testing. We have some hundred images taken with high resolution (5472*3648 pixel) cameras. The used camera model is the OcamCalib model (with 16 distortion coefficients). We want to do some training with neural networks, but almost all neural networks use Pinhole models (rarely fisheye, never OcamCalib), so we have to calculate the parameters of a fisheye lense that matches our OcamCalib model. We can do that with this repository.

I used your software and used the KB, RadTan and UCM models as target model. The KB did not really work. The resulting image had a large black circle in the middle, probably because it is not good enough to capture the details of the OcamCalib model. RadTan and UCM worked, though. They just have noise (specific patterns) scattered over the image.

Do these artifacts occur because RadTan and UCM are less accurate models? (I know that I can use the original images and use the calculated intrinsics and pretend that the original images have been taken with the hypothetical camera, I just want to know if that's the reason)

I can't upload the original image. It exceeds githubs file size limit.

[eowjd0512]

The black artifacts appearing when restoring images using newly extracted parameters are simply due to a lack of interpolation.

If you look at 'prepare_data()' for recovering the image, it merely creates the image by unprojecting all points in the image to 3D and then projecting them back using the estimated model.

Furthermore, as seen in these lines, the artifacts can occur because the projected points were directly converted to integers without considering their precision.

I am aware of this issue, but since image recovery is intended for reference purposes only, I have decided to overlook it as a known issue.

To restore the image without any artifacts, instead of interpolating from 3D to 2D projection, I considered implementing an inverse warping method that generates an unprojected ray from every pixel in 2D using the estimated parameters, and finds the nearest 3D point for each ray. However, this would increase the dependency on additional libraries, and since the previous method is sufficient, I decided not to proceed. (For reference, see the deleted function 'recover_image()' in commit f93740297b0c86e47d081fa0eaec36c134ce1785.)

The key metric you should closely monitor for accuracy is the reprojection error.

Therefore, as you know, you should continue using the estimated parameters as they are.

[eowjd0512]

By the way, the conversion of projected points from float to integer was fixed in commit c62571ff19732dab89e80af30c35e587c07e3401. Could you test it out?

[Sheradil]

I will do so after the weekend. Thank you for your detailed and fast response.

[eowjd0512]

@Sheradil I have a few issues to discuss regarding the OCamCalib model.

1. Unprojection: I am not very familiar with the OCamCalib model, so I followed the tutorial provided at Scarabotix OCamCalib Toolbox for MATLAB. It seems that the unprojection parameters were negative.

   

   (The image above shows the parameter results obtained through the MATLAB version of the OCamCalib toolbox. The parameter marked by dragging is suspected to be negative, which is used for the cam2world, that is, the parameter used for unprojection.)

   I have seen a few systems using the OCamCalib model where the unprojection (cam2world) operation involves negative z-values:

   • ocamcalib_undistort
   • MultiCol-SLAM
   • OCamCalib Google Group

   I was not convinced about why a negative value was necessary, so I did not include it in the formula. Instead, I multiplied the unprojection coefficients in the example dataset I provided by a negative number because I also observed correct parameter estimation when using negative values in my experiments.
   (If you look at dataset/OcamCalib/OcamCalib.yaml, you can see that it provides values for unprojection taken as negative from the MATLAB results, unprojection: [2.093037e+02, 0.000000e+00, -2.216175e-03, 5.122558e-06, -2.149945e-08].)

   If taking a negative for z in the formula is normal for the OCamCalib model’s cam2world, I am considering making this adjustment. Do you have any insights on this?

   (You might also try applying a negative to the input parameters and testing it yourself. If this issue occurs frequently, I think it would be wise to proceed with implementing a negative in the internal code.)

2. Number of Unprojection Coefficients: As per OCamCalib documentation, I have fixed it to a 4th degree polynomial (5 coefficients). Hence, there are 5 unprojection coefficients in this system, and you should check if this matches your parameters. If it differs, please let me know, and I will create an appropriate branch.

3. Number of Projection Coefficients: Similarly, the number of coefficients for projection is fixed at 12. This number was chosen based on experimental sufficiency. According to OCamCalib, the polynomial order for projection is automatically determined to minimize error; here, I specified 12 as the maximum number. The number of projection coefficients is used solely for projection and does not impact the conversion from OCamCalib to other models.

Please check if these issues align with the system you are implementing and let me know after careful consideration. Particularly for issue number 1, I need to decide whether to implement taking a negative within the code without changing the input.

[Sheradil]

By the way, the conversion of projected points from float to integer was fixed in commit c62571f. Could you test it out?

I just tried the commit https://github.com/eowjd0512/fisheye-calib-adapter/commit/c62571ff19732dab89e80af30c35e587c07e3401 and it did not work. I am not sure if it made things better or worse. I will take a close look at it when I find a bit more time.

My issues might have another reason, though: The Ocam calib images we use here are taken with a NavVis scanner (the scanner has 4 attached cameras). NavVis is probably not as well known as e.g. Nvidia, but their scanners can create highly accurate point clouds. Their calibration parameters contain 18 projection and 5 unprojection parameters. I to changed MAX_POLY_NUM to 18 and tested it again, but I still have these issues. It might be related to some changes they made to the ocam model, they inverted the z axis and changed x and y axis. But I am not sure if those changes are relevant. Most likely not.

I will make the image available on another page for you (tomorrow), so that you can see for yourself.

[eowjd0512]

Once again, the output image is only for reference, and I did not thoroughly consider interpolation. It is sufficient if the pixels from the input image are near their appropriate locations in the output image. You can directly use the estimated output model parameters in your network.

[Sheradil]

Found a place to upload images that are large. https://fastupload.io/6e0057d2019d1483 OcamCalib.txt I understood that it is not needed, just in case you are interested. The text file obviously has to be renamed to a yml file. But github does not allow these in comments.