Reprojecting disparity causes grid artifacts.

[xhchen10]

Thanks for your impressive work. I am working on rectifying the left event camera and right frame camera and reproject disparity to the view of my rectified version. The first rectification step is well rectified thanks to the accurate intrinsic and extrinsic camera matrices. However, when I reproject the official ly released disparity maps, which are aligned with camRect0 (rectified left event camera), to the view of my rectified version, I find obvious grid artifacts on the reprojected disparity map, seen in the following figure. Could you kindly help me figure out any reason or solution? Thank you so much.

[magehrig]

The way you should reproject your disparity map into the other camera view is by projecting the points into 3D and the projecting them back into your other view. You shouldn't get any artifacts this way. Have you done the projection of the disparity map via 3D projection?

Overall, I am not sure what you would like to achieve. The disparity map is available in camRect0 (which is the rectified view of the left event camera, as you mentioned). What is your target view into which you want to map the disparity?

[xhchen10]

Here are the steps I've conducted. 1) Project the 2D points into 3D by cv2.reprojectImageTo3D; 2) Project the 3D points into my other view. 3) Project the 3D points into 2D with my new rectified intrinsic matrix. I think it is consistent with what you describe.

I would like to conduct stereo matching between the left event camera and the right frame camera, which requires extra rectification and disparity projection.

[magehrig]

What you describe makes sense. My hypothesis of why the pattern exists is that it's a small artifact from the camera distortion. We do not have it in the provided disparity maps because I directly used the pointclouds in 3D from the lidar. In your case, however, the 3D points are slightly different because I did nearest-neighbor assignments (in pixel space) of the projected 3D points to generate the disparity maps. Now this creates a small bias conditioned on the distortion pattern from the camera in which the pointcloud was projected. So by projecting it into the other view, most likely you see some artifacts from this nearest-neighbor assignment. Anyway, I don't think this is a problem at all. That amounts to a subpixel error which is anyway less than the actual error in aligning camera and lidar. If you want to minimize this effect, you can project the disparity maps from the left frame camera which has higher resolution.

Hope that helps

[magehrig]

I think what is more important is that you create an overlay of your other view with the disparity map similar to how it was done here: https://github.com/uzh-rpg/DSEC/issues/14

Just to verify that the alignment looks correct

[xhchen10]

What you describe makes sense. My hypothesis of why the pattern exists is that it's a small artifact from the camera distortion. We do not have it in the provided disparity maps because I directly used the pointclouds in 3D from the lidar. In your case, however, the 3D points are slightly different because I did nearest-neighbor assignments (in pixel space) of the projected 3D points to generate the disparity maps. Now this creates a small bias conditioned on the distortion pattern from the camera in which the pointcloud was projected. So by projecting it into the other view, most likely you see some artifacts from this nearest-neighbor assignment. Anyway, I don't think this is a problem at all. That amounts to a subpixel error which is anyway less than the actual error in aligning camera and lidar. If you want to minimize this effect, you can project the disparity maps from the left frame camera which has higher resolution.

Hope that helps

Thank you so much for the comprehensive analysis and the effective solution. The grid artifacts are minimized with the higher resolution disparity maps.

[xhchen10]

I think what is more important is that you create an overlay of your other view with the disparity map similar to how it was done here: #14

Just to verify that the alignment looks correct

The reprojected disparity map is aligned with my other view as expected. Thanks for the kind reminder.