Z Scaling and Culling

[BarusXXX]

The Z scaling of the generated depth map has wrong proportions in relation to the size of the X Y, @mariuszhermansdorfer mentioned that what is coming out of the camera should be in mm so if the value in the image ushort cast to int is 200 this would mean that the object is 200mm away from the camera? The values I am getting from the depth camera seem to have the wrong z scaling.

As regards to culling, @mariuszhermansdorfer how had you avoided getting cascade effect (due to sudden change in z see above) if some of the objective points have Z=0. Or you you simply show the scene from the top?

[BarusXXX]

For NFOV the size of the mesh is 1532 wide and 1272 height For WFOV the size of the mesh is 3460 wide and 3460 height Sensor height does not change the size of the pixels (It should not)

Hexagonal capture area creates 4 corner triangle occlusion zones that have no data? - think this is changed when transformed to color camera. In these 4 corners it seems like it copies the last measured edge Z all the way to the edge of these no data areas.

[mariuszhermansdorfer]

@BarusXXX, regarding cascading: this is what the trimming portion of the component is meant to do. Trim the unneeded columns/rows from left/right and top/bottom.

Kinect has a minimum and maximum distance that it can read. If a pixel falls outside of these bounds (or there is an invalid pixel for other reasons), it defaults to 0. Here we make sure to only take pixels, which are above 200 mm away from the sensor: https://github.com/mariuszhermansdorfer/SandWorm/blob/2b53d409f3cc18aa32da5d14bd61c5a1802dead8/SandWorm/Components/BaseKinectComponent.cs#L136

and here we are trying to capture random 0 values and replace these with a value from their neighbor: https://github.com/mariuszhermansdorfer/SandWorm/blob/2b53d409f3cc18aa32da5d14bd61c5a1802dead8/SandWorm/Components/BaseKinectComponent.cs#L142

Regarding various FOV of Kinect Azure. Since we are interested in a rectangular sandbox, we need to trim away the unneeded portion of the image anyway. The only thing to check here would be, whether these FOV still produce a rectangular array of pixels, or does it have varying amount of columns in subsequent rows. I hope it's the former, otherwise we would have to transform it somehow.

[BarusXXX]

Here you can see a mug about 350mm away from my sensor and my hand at about 250mm clearly the z is underscaled (the mug looks flat not round) or the pixel size is over scaled.

[mariuszhermansdorfer]

Can you print a few values from the original depth frame to confirm that these are indeed in millimetres? This was true for KV2, hence we are using the units multiplier here:

https://github.com/mariuszhermansdorfer/SandWorm/blob/2b53d409f3cc18aa32da5d14bd61c5a1802dead8/SandWorm/Components/BaseKinectComponent.cs#L185

Another reason for this might be that the calibration component resulted with some data and the main component is now using it as a base. Again: the calibration component assumes, that is sees an empty table and memorizes all the values it can see. Then it is subtracting these from the 'live' image. If you ran a calibration process while the mug was visible by the sensor, it will be flattened out.

[BarusXXX]

717mm is spot on my table (i set the kinect to the edge of the table and measured distance to floor) height so we are right in assuming they are mm. What I do not get is the initial and trailing 135 indices full of zeros and a regular pattern of zeroes in the data. My guess is that these are the unmeasured corners of the rectangular shape. So the 0 gaps must be the corner edge of the rectangle that is unmeasured. (by the way this is NFOV)

[BarusXXX]

and here we are trying to capture random 0 values and replace these with a value from their neighbor: https://github.com/mariuszhermansdorfer/SandWorm/blob/2b53d409f3cc18aa32da5d14bd61c5a1802dead8/SandWorm/Components/BaseKinectComponent.cs#L142

This explains the streaking to right screen.

It also explains; Because we always take the left neighbor by default and the sensor has occluded corners bound to 0 without the transformation, the first line of zeros closest to the origin will be 0 while the rest will default from the last value, this should be improved to getting the neighbor from both sides and averaging the non zero values of the two.