Questions about the depth calculation

[felja633]

I haven't found any detailed technical documentation about the Kinect v2 and I really want to know how the depth calculation works in libfreenect2 so my questions are:

1. According to the article "The Xbox One System on a Chip and Kinect Sensor (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6756701)" the Tof-system uses differential pixels and I see from the code that 9 images are processed. What are these images exactly and what kind of processing is made on the chip?
2. From the code I see that three phases are calculated from the 9 images how are those three phases combined. I can't really follow the algorithm from the code so how does it work?

[floe]

As a very rough summary, the 9 images are divided into three groups. Within each group, the images correspond to phase-shifts of the modulation signal by 0°, 120° and 240°. Each of the 3 image groups is collected at a different modulation frequency to deal with ambiguous distance measurements. That's about all I know :-)

[felja633]

Thanks! That's my interpretation of the code and the article referred to in the question, nice to have that confirmed

[xlz]

But what exactly does the 10th image do (352*424*2 bytes)? Its purpose is definitely not to waste bandwidth.

[JoshBlake]

The 10th image is reportedly for use in Xbox to track the IR blinkers (each with unique frequencies) in the Xbox controllers. This is how the Xbox One could track the controller-player relationship.

[xlz]

A tenth infrared frame without active illumination is recorded to adjust for ambient brightness independent of the active illumination.

From J. Stühmer et al, Model-Based Tracking at 300Hz using Raw Time-of-Flight Observations, ICCV'15.

Presumably the tenth frame is indeed used for infrared brightness normalization?

@sh0 @floe @JoshBlake

[JoshBlake]

Edit: Sorry I already said that in this thread a year ago, didn't read up enough!

I haven't heard about using it for normalizing the brightness, although that is plausible. My information has the tenth frame being using for the Xbox One controller tracking. Xbox One controllers emit a unique IR frequency from the top/front, and games can use this to track e.g. when players switch positions or swap controllers, and coordinate that with the regular skeleton tracking.