The value from getCameraIntrinsics() is constant, but the image is offset

[saulthu]

I am relying on dai::CalibrationHandler::getCameraIntrinsics() to provide factory calibrated intrinsics. The problem I'm seeing is that if I configure that pipeline to output THE_1080_P I get the exact same calibration values as when I configure the pipeline to output THE_4_K but with the ISP downscaling to 1080p -- that in itself is not a problem, but visual inspection of the image shows there is a true shift in the image contents that is not reflected in the calibration parameters. It is not clear to me when the returned calibration is valid, and when it is not valid.

• Is getCameraIntrinsics expected to return different values depending on the configured pipeline?
• If not, then which pipeline configuration was used to create the calibration?
• If calibration was performed on THE_1080_P, for example, is there a way I can reliably convert that to an ISP-scaled version of THE_4_K?

This is captured using THE_1080_P:

This is captured using THE_4_K but scaled down by ISP to 1080p:

I'm seeing a significant vertical shift in the image when using THE_1080_P (left) vs using THE_4_K (right):

Overlaying the two images in Gimp and setting layer mode to Grain extract we get this:

Zoomed on the whiteboard text:

Moving the layer from THE_4_K down 6 pixels and left 1 pixel gives a good alignment:

Toggling between overlaid, aligned images: (Also note the significantly lower image quality when using THE_1080_P vs higher quality from THE_4_K. They both have luma and chroma denoise set to zero. I'm guessing that THE_1080_P has these settings permanently turned on to some level?)

Both pipelines give the exact same result from getCameraIntrinsics:

[[1142.2,      0, 983.7],
 [     0, 1142.2, 521.4],
 [     0,      0,     1]]

So, there is no [-1, 6] pixel shift as would be expected in the calibrated center point, if the pipeline were taken into account by this function.

I get the intrinsics by reading the calibration. After opening the device with the given pipeline I use cal = dev->readCalibration() and later use cal->getCameraIntrinsics(dai::CameraBoardSocket::RGB, 1920, 1080).

[saulthu]

A possible answer to the question of what pipeline configuration is used during factory calibration

I notice there's a Luxonis calibration repo that appears to be for factory calibration:

https://github.com/luxonis/Factory-calibration-DepthAI

If this is the code used for calibration then it appears that the IMX378 cameras, for example, are calibrated with resolution set to THE_4_K, (as seen here).

So, this would suggest that I should be using THE_4_K and downscaling it using ISP to get a correct factory calibration. It also suggests that it is THE_1080_P configuration will be wrong due to a [-1, 6] pixel shift relative to THE_4_K configuration downscaled to 1080p.

[saulthu]

This does, however, leave me with a concern regarding the validity of the scaled calibration values that are output by CalibrationHandler::getCameraIntrinsics.

At the very least, the method used by CalibrationHandler appears to be slightly incorrect. The pixel coordinates used by OpenCV calibrations have origin [0, 0] at the center of the top-left pixel, so the image actually extends from [-0.5, -0.5] to [W-0.5, H-0.5], and applying a scale to this will effectively offset the calibrated center away from the true principal point. This is a small thing.

My larger concern is whether the ISP introduces other pixel coordinate shifting, which I haven't yet looked into.

[saulthu]

As for the ISP introducing image shift, there does appear to be a small amount of shift introduced, but appears to be less than 1.0 pixel.