Body tracking is too slow and inaccurate

[rfilkov]

Describe the bug

By all means Azure Kinect is the best Kinect so far, and will be probably the best depth sensor on the market. The sensor SDK is pretty stable and good, providing almost everything an average user would want. But the body tracking subsystem is ruining this positive user experience. In means of API this SDK is great too, but the DNN model performance is much worse than the body tracking of Kinect-v2. The joint positions are inaccurate by fast movements. The body index map is not very accurate, as well. It does not fully match the user's silhouette on the depth frame. On my GTX 1060 it takes 2-3 depth frame cycles to process a body frame. Hence, it works at about 10 fps.

To Reproduce

1. Run Azure Kinect Body Tracking Viewer.
2. Stand in front of the sensor.
3. Make fast arm movements.
4. Look at the arm joint positions with regard to the real arms.
5. Look at the colorized body-index map with regard to the real body.

Expected behavior

1. I expect the body tracking to work at least at 30 fps or more, i.e. faster than the depth frames arrive.
2. I expect the body joint positions to match as precisely as possible the user's joints on the depth frame.
3. I expect the body index map to match as precisely as possible the user's silhouette on the depth frame.
4. I expect the body tracking to be less demanding, in means of hardware and 3rd party software requirements. GTX 1070 + CUDA + cuDNN + manually setting paths would be too much for the average user.

Please consider at least providing some option to the users, who don't have high end graphics cards and would like to get Kinect body tracking out of the box, without (or with minimum) extra installations. As far as I remember, Kinect-v2 used random forest model(s) for its body tracking. The performance was great and no extra installations were needed, back then in 2013/14.

Logs

Screenshots

Desktop (please complete the following information):

• OS with Version: Windows 1809
• Kinect SDK Version: 1.1.0
• Body tracking SDK Version: 0.9.0

Additional context

I believe most Kinect users would expect better, more accurate and more performant body tracking experience, not worse. And now, with Apple adding people segmentation and body tracking to their AR-Kit 3.0 I would expect Kinect (with all these years of experience) to provide a better user experience in all aspects than anybody else.

[vpenades]

If you are interested, Nuitrack currently supports Kinect Azure and provides fast CPU-only skeletal tracking with it.

Thanks for the tip. I'm aware of nuitrack and I've been following their progress for a while now.... unfortunately, our use case prevents us from using NuiTrack for a number of reasons.

Long story short: our customers demand low end, easy to use, easy to install, no headaches, no power hungry, no internet required, and easy to purchase, solutions. The Kinect2 met almost all these requirements. Everything that came afterwards for one reason or another, falls short from these requirements.

Fortunately for us, we're progressing on solutions that only require a webcam, it's not as good as the Kinect2, but we've managed to fit our requirements with it.