Disparity results in incorrect point cloud

[darren-harton]

I've noticed that when I get disparity values back from the Bottlenose camera, running cv2.reprojectImageTo3D results in a point cloud that is about 40% too large. Otherwise, the points look correct.

To test, I tried calculating the disparity on my own by requesting raw images from the camera. In this case the resulting point cloud is scaled correctly.

In addition, I've tried comparing rectified images coming off of the camera to images that I've rectified myself. What I found is that the images are basically identical except that they are shifted relative to each other. The offset is about 10 x -3 pixels.

Firmware: DeviceModelName: BN_4K_4GB_320MB_CS_ST DeviceVersion: 0.4 DeviceFirmwareVersion: v0.2.132

calibration.yaml

cam0:
 fx: 1322.477467104818
 fy: 1323.751092313471
 cx: 949.786919379760
 cy: 555.350700278323

 k1: -0.266834378242
 k2: 0.155438140035
 p1: -0.000309996016
 p2: 0.000047213813
 k3: -0.054694611579

 tvec: [0.000000000000, 0.000000000000, 0.000000000000]
 rvec: [0.000000000000, 0.000000000000, 0.000000000000]

 width: 1920
 height: 1080

cam1:
 fx: 1320.753362065004
 fy: 1321.894620230395
 cx: 969.575226893761
 cy: 520.765496143764

 k1: -0.269198805094
 k2: 0.164478644729
 p1: 0.000136121511
 p2: 0.000105676147
 k3: -0.063760466874

 tvec: [-0.133988077477, -0.000193804114, -0.000745757388]
 rvec: [-0.000855856506, 0.001549896790, -0.001169957099]

 width: 1920
 height: 1080

sample_imgs.zip

[treideme]

@darren-harton Can you share sample code and data/images for the points as well, that we can use as basis for this? Further, can you check what you used as offset (or default?) for the cameras for the calibration, see here: https://docs.labforge.ca/docs/calibration -> "Image Offset"?

[yrizwan]

@darren-harton can you also please see if you are using a minimum disparity setting? We may have found a bug where the final disparity output does not add this minimum disparity. E.g. if min disparity is 100 and its searching for 128 disparities, If it finds a match at say 111, the disparity image should say 211, not 111.

[darren-harton]

@treideme I've attached a jupyter notebook (and supporting scripts) that reproduces the issue using saved images. Let me know if you need anything else. The image offsets haven't been changed from their default values.

@yrizwan The min disparity is set to zero.

Thanks for looking into this!

scaling_issue_notebook.zip

[darren-harton]

I just realized I left a confusing comment in that notebook. Here's a corrected version:

The camera is should be just over under 2 meters from the chessboard.

[yrizwan]

Hello @darren-harton, your notebook was very helpful. We were able to trace the issue to a rectification bug inside the camera. We will be issuing a new FW that fixes this, once done. Thank you very much!

[treideme]

@darren-harton Can you please update your firmware and Bottlenose utilities to this release.

https://github.com/labforge/bottlenose/releases/tag/v0.2.134

We fixed some scale issues in the firmware. Also please look at our updated stereo calibration documentation.

Let us know if this works.

[darren-harton]

Thanks a lot for your help.

I recalibrated referencing the updated documentation and ended up setting OffsetY1=378 instead of leaving it at the default 440. The point cloud appear to be at the correct scale, but now the depth values are no longer aligned with the rectified left image.

I tried shifting the image using the OffsetX1/Y1 values, but that didn't look right either.

data.zip

[yrizwan]

Hi @darren-harton, we tried recreating this but ours are aligned to the left image. We also passed your rectified images through the camera's test procedure and they came out aligned as well. For good measure, we tried to manipulate the X/Y offsets post-calibration, and that did not recreate this issue either (note for readers, please don't change these offsets post calibration, and make sure to re-enter these values every time the camera is booted).

Is it possible the camera moved between the disparity capture and the raw captures? We noticed the timestamp on the disparity image is 5 minutes earlier. Maybe someone kicked the tripod?

Notice how in our test result the top right cabinet still visible:

[yrizwan]

Hi @darren-harton, was this helpful to you? Can we provide you any further assistance?