Truncated z values in results from mrcal.stereo_unproject

[akonneker]

If I use mrcal.stereo_unproject with floating point disparity values generated from a recent ML-based stereo approach, the output z values of the point cloud are truncated, almost like I'm using integer disparity values from something like openCV's StereoSGBM.

Do you know why this is the case? If I look at the unique z-values in the points, you can see that some sort of truncation is happening. If I do the disparity->depth calculation manually, I get something more along the lines of what I would expect. Here is a sample snippet, with output.

disparity = np.load(data_path)

models = [mrcal.cameramodel(model_path) for model_path in model_paths]

models_rectified = \
    mrcal.rectified_system(models, az_fov_deg=72, el_fov_deg=48, rectification_model="LENSMODEL_PINHOLE")

xyz = mrcal.stereo_unproject(-disparity.astype(np.float32), models_rectified)
print("cloud shape", xyz.shape)

xyz2d = xyz.reshape(-1,3)
print(xyz2d.shape)

print("unique disp values, mrcal:", np.unique(disparity).shape)
print("unique x values, mrcal:", np.unique(xyz2d[:,0]).shape)
print("unique y values, mrcal:", np.unique(xyz2d[:,1]).shape)
print("unique z values, mrcal:", np.unique(xyz2d[:,2]).shape)

lensmodel,intrinsics_data = models_rectified[0].intrinsics()

#print(models_rectified[1])

f_x = intrinsics_data[0]
baseline = models_rectified[1].extrinsics_Rt_fromref()[3,0]

Z = (f_x * baseline) / disparity
print("unique z values, manual:", np.unique(Z).shape)

cloud shape (793, 1298, 3)
(1029314, 3)
unique disp values, mrcal: (962187,)
unique x values, mrcal: (249765,)
unique y values, mrcal: (132035,)
unique z values, mrcal: (1876,)
unique z values, manual: (954228,)

[dkogan]

Hello. Would it be possible to share the data as well, so that I could reproduce the issue here? Would make it unnecessary to guess about what's going on

[dkogan]

If I WERE to guess I'd say the difference is that you're giving mrcal 32-bit disparities, but checking with 64-bit disparities. What if you compute Z with 32-bit disparities too?

But that's just a guess. If it's wrong, send me the data

[akonneker]

The truncation isn't a float32 vs float64 thing. It's on the order of if you truncated disparity to integer values, but not quite that bad. Here is the data for you to poke at. It has the rectified images, the disparity rendered as a png and an npy file, and the camera models from mrcal (opencv8 model).

sample_data.zip

[dkogan]

Thanks for sending that over. This isn't a "bug", but rather a surprising behavior if you do things in unexpected ways. The questionable code: https://github.com/dkogan/mrcal/blob/ef439a17eb7003a49dd6f85744976b25cf54686c/mrcal/stereo.py#L1261

Note that the "sparse" stereo path (if you specify the pixels you care about explicitly) uses float to represent disparities. But the "dense" path (if you want to process the whole image; what you're doing) uses np.uint16. This was written this way because every disparity-computing routine I've ever seen reports its results in a 16-bit integer type. In your case, this results in rounding each disparity down to the next lower integer.

At some point in the near future I'm going to fix this; haven't decided exactly how yet. At the very least there should be documentation. For your immediate usage, I would do this:

disparity_scale = 64
mrcal.stereo_unproject(-disparity * disparity_scale,
                       models_rectified,
                       disparity_scale = disparity_scale)

This mimics how traditional disparity search routines output their results: as 16-bit integers, but in sub-pixel units.

Let me know if this doesn't solve your issue