Low accuracy pointcloud compare to realsense

[BartvanMarrewijk]

Describe the bug I am imaging a plant from a height of approximately 66 cm. The resulting pointcloud of the my Oak-d S2 camera is really noisy. I compared with a realsense D415 and the pointcloud of the realsense was much better.

Oak-d S2 topview; D415 topview;

Oak-d S2 sideview; D415 sideview;

Minimal Reproducible Example Append the MRE to the bug report, instructions here I am ussing following code to create the pointcloud: https://github.com/luxonis/depthai-experiments/tree/master/gen2-camera-demo depth resolution is set to THE_800_P Other settings:

lrcheck = True # Better handling for occlusions extended = True # Closer-in minimum depth, disparity range is doubled subpixel = False # Better accuracy for longer distance, fractional disparity 32-levels median = dai.StereoDepthProperties.MedianFilter.KERNEL_7x7

I have the feeling that something is incorrect with my camera, however, it is brand new and the calibration should be as expected. I am not sure what I am doing wrong, but there must be something considering the pointcloud quality.

[Erol444]

Hi @studentWUR , It's a bit of an unfair comparison when comparing passive and active stereo. A better comparison would be to compare Pro against Realsense, and people have already done this, eg. on this video, pcl quality shown:

And it's clear the passive stereo struggles at some texture, while active stereo doesn't. Also note that subpixel mode will be better in your case as well, as it will have "less layering". Thoughts? Thanks, Erik

[BartvanMarrewijk]

Dear Erik, Thank you for your reply. The laser was turned off for the realsense, so the comparison is to my opinion fair. The subpixel mode seems to improve the issue. Depending on the object of interest.

[BartvanMarrewijk]

Sorry to open this one again. I have two questions:

1. I know have some issues/problems with the accuracy in X & Y direction. Y direction measured = 0.372 [m], actual = 0.39 [m]

X direction measured = 0.561 [m], actual = 0.570 [m]

Especially, the error in the y direction worries me a bit. I again used, the same code as in this repo: https://github.com/luxonis/depthai-experiments/tree/master/gen2-camera-demo depth resolution is set to THE_720_P, rgb resolution is set to THE_4000X3000

Is this the accuracy to be expected or a calibration issue?

For getting the intrinsics, it uses, which results in w=4000, h=3000 w, h = camRgb.getIspSize() intrinsics = calibData.getCameraIntrinsics(dai.CameraBoardSocket.RGB, dai.Size2f(w, h)) The saved RGB images is actually 4056x3040, but depth image is 4000x3000. Is it actually correct to use the intrinsics corresponding to 4000. I tested both, but both have similar inaccuracy as in the image above.

[Erol444]

Hi @studentWUR , What's the Z error? As X and Y are just calculated from the Z. Sharing the pointcloud would also help.

[BartvanMarrewijk]

Thank you for your fast response. The Z error is minimal, Z direction measured = 0.806 [m], actual = 0.807 [m] I would expect that an error in the Z, would result in an error for both X and Y with the same magnitude.
The pcd was too big for git so I have to send a link. You can download the files here:

https://filesender.surf.nl/?s=download&token=a46ee1d1-ceff-4551-a42e-ea4b64223dd6

[Erol444]

Hi @studentWUR , Are you using rgb-depth alignment? looking at the code, it takes intrinsics from the right camera, but I believe that if rgb-depth alignment is enabled it should take the ones from the color cam. Thanks, Erik

[BartvanMarrewijk]

Hi @Erol444,

I am indeed using the rgb-depth alignment, so yes I use the intrinsics of the color camera. I only used the intrisincis calculated with (w=4000, h=3000), instead of the (w=4056, h=3040): color_int = calibData.getCameraIntrinsics(dai.CameraBoardSocket.RGB, dai.Size2f(w, h)) Thanks for diving into it :)

[Erol444]

Hi @studentWUR , Does it work with 4k or 1080P color resolution (and 4k or 1080P camera intrinsics)? As I believe we do calibration on 4k images (not 12mp), which could potentially be the source of the problem. Thanks, Erik

[BartvanMarrewijk]

I have created the pc for both resolutions (1080p and 4k). Then the error in y direction is minimal, but there is a large offset in x direction. Does this mean that it is a calibration issue? Y direction measured = 0.39 [m], actual = 0.39 [m] X direction measured = 0.59 [m], actual = 0.57 [m]

You can find the files here: https://filesender.surf.nl/?s=download&token=267454f5-05b7-4374-bae4-5486ac5094c8

[themarpe]

CC: @Erol444 @HyChT

[KAkikakuFukuhara]

Hi I think it's because of lens distortion.

[BartvanMarrewijk]

@KAkikakuFukuhara, indeed it seems that the image is slightly distorted looking at the lines of the object. That would mean that the calibration is inaccurate. However, this device is completly new and factory calibrated. Given the specs of the camera I think that this error is too large.

[KAkikakuFukuhara]

The https://github.com/luxonis/depthai-experiments/tree/master/gen2-camera-demo program uses Open3d.geomety.PointCloud.create_from_rgbd_image to generate the point cloud. It does not use lens distortion. However, this issue may not be relevant, as the drawing tools appear to be different when looking at the diagrams provided.

[BartvanMarrewijk]

I used indeed that software to create the pointcloud. The drawing tools above are from cloudcompare. But I think you are right, it has todo with the distortion and maybe also with depth to rgb alignment when using an AF lens. See another example below, of a flat object.

So the question is how to make these point clouds correct when using an AF lens. @Erol444 could you have a look.

[themarpe]

@studentWUR WRT AF, it has to be set to the lensPosition as specified in calibration, to achieve same mapping between depth and RGB imager

[BartvanMarrewijk]

@themarpe, okay clear, but how to get the used auto focus at the factory calibration? If I am running the default example to get the parameters, it doesnot return anything about the focus at the moment of calibration. So, what was the auto-focus during factory calibration?

[BartvanMarrewijk]

Small update, so I saw this issue onf depthai-core: https://github.com/luxonis/depthai-core/issues/463

So with my oakd pro I made three points clouds, with three different auto-focus 80,130 (as in issue 463), and 255. All of these point clouds are visible below; So regardless the AF setting, the distortion is still visible. @themarpe any ideas?

[Erol444]

cc also @CenekAlbl

[CenekAlbl]

We are actively working on this and it seems it will be fixed by an improved calibration. We will keep you posted.

[doisyg]

Any update on this ? We see the same distortions on factory calibrated cameras and without RGB on OAK Pro D Wide

[CenekAlbl]

There is no ETA on this yet, but when we figure it out a recalibration will be needed. In the meantime, have you tried to calibrate the camera using calibrate.py in the repo? Is it possible for you at all to print a calibration pattern and do the calibration? This would also be of great help to us to debug.

[HyChT]

Hi @studentWUR, while waiting for us to upgrade the calibration procedure, I believe you can try to do manual calibration. This depth plane curvature can be caused by the distortion from the Mono images. AFAIK, this is due to the number of the power of Brown Conrady un-distortion coefficients. By default, we are using k1 to k6, which can be overkill/overfit the distortion map. You can try using only k1 to k4 or perhaps k1 to k3 and leave others as 0.

For testing purposes, you can use pipeline.setCalibration to override the calibration data. You can use the calibration example here: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html And this for rectification: https://docs.opencv.org/4.8.0/d9/d0c/group__calib3d.html#ga617b1685d4059c6040827800e72ad2b6

[BartvanMarrewijk]

Hi @CenekAlbl and @HyChT,

Thank for the solutions. To be honest, based on experiences with realsense, calibrating any device manually is mostly an error prone solution. Since there is no standard produce (like in a factory), each manual calibration will result in different checkerboard poses resulting in different calibration results. so I wonder if my calibration will result in a higher accuracy then factory calibration. But I will have look.

Secondly, is there any option to include the distortion coefficients to create the point cloud as well?

[BartvanMarrewijk]

@CenekAlbl and @HyChT, any update on the improved calibration?

[Erol444]

Hi @studentWUR , we apologize for the delay. You can find updated calibration docs here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration/ Kindly let is know whether the updated calibration procedure has improved your calibration. Thanks, Erik

[BartvanMarrewijk]

Dear Erol, I have tested it now for one camera, at first my point clouds were completely empty and now they I have a point cloud. I still have todo validate the accuracy after calibrating. I will keep you updated.

By the way, in the calibration.py I found that the active focus is still active. Mostly I am using this cameras on a approximate distance of 1m. Would it be better to calibrate with a fixed lens position? What is your experience in this?

[Erol444]

Hi @studentWUR , I believe AF will wait until focus converges, and then take that value and set it manually, so it won't move during the calibration. I don't think it's needed, as it should get the best focus for your distance using this process.