search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.5k stars 620 forks source link

Release a V2 with integrated body tracking #1253

Open Chris45215 opened 4 years ago

Chris45215 commented 4 years ago

In light of the long-running thread about body tracking latency, and the difficulties of isolating the latency, as well as the debate over prior body tracking algorithms and the difficulty of using a GPU in every system, the only complete solution is to integrate the CUDA cores and some SDRAM into the sensor. This would allow direct communication between the sensors internal processor and the GPU, without the delay of routing through the computer's CPU, and that factor alone will reduce latency. Further, it will allow much simpler (I hope) debugging of the latency issue. And, this will finally enable the 'killer app' feature of the sensors: multi-sensor body tracking. Currently a high-end computer can handle 2 or 3 sensors at most, but if body tracking is built into the sensor then the PC resources are free and the USB bandwidth is massively reduced, so a single PC can easily host the 10 sensors that can be synchronized. It will also allow the body tracking to be used in small form factor PCs, far more IoT applications (there are very few IoT PCs with GPUs), and game consoles that use AMD GPUs (like Microsoft's own xBox). Further, it would make the body tracking compatible with the HoloLens headset, which has much lower latency head and hand tracking than the Azure Kinect can manage, so that would allow the headset to work with full body tracking applications (without a special mocap suit or Vive pucks). It would also remove resource conflicts, such as the inability to use Microsoft Remote Desktop or Google Hangouts while running body tracking (seriously, try running either of those then launch Body Tracking Viewer, it will crash).

This is not too extravagant of a request, as much of the cost of the required GPU is due to the onboard RAM and video output, while the DNN for body tracking only uses about 1GB of the RAM and needs no video output. This also wouldn't create an unreasonable power or thermal budget - the sensor already requires an external power supply, so a V2 with CUDA cores would just require a larger one. If this would make the device too large, I suggest removing some or all of the microphones, as there is little use case for a 7-microphone array.

qm13 commented 4 years ago

@Chris45215 love the idea. Would you mind also opening it on the Feedback Forum https://feedback.azure.com/forums/920053-azure-kinect-dk. Microsoft pays a lot of attention to Feedback Forum and how the developer community perceives ideas. The team has grand plans for Azure Kinect and Body Tracking some of which will be made public soon. Just remember that Rome was not built in one day.

rfilkov commented 4 years ago

@Chris45215 , if you ever post this idea to the feedback forum, please share the link here.

mrcfschr commented 4 years ago

I love the idea too. I linked it here: https://feedback.azure.com/forums/920053-azure-kinect-dk/suggestions/41021185-please-get-body-tracking-to-work-on-hololens2

vpenades commented 4 years ago

Certainly, having processing power on the sensor would solve a lot of problems... but given that the Kinect Azure is still in the begining of its market lifetime (will see how long that lifetime will be)... I don't think we will see a new sensor in less than 5 years... and if the kinect azure proves to be a failure... I don't think we will ever see another one from microsoft.

So we need solutions for the current sensor. The hardware is great, the software side not so... and it would be a shame if the kinect azure sensor becomes a flop just because the sofrware side was not up to the hardware.

Chris45215 commented 4 years ago

Everyone, I just posted the suggestion to the feedback page on https://feedback.azure.com/forums/920053-azure-kinect-dk/suggestions/41032135-release-a-v2-with-integrated-body-tracking, and tagging @rfilkov and upvoted @marcfischer123467 's suggestion as well. I would love to see VR integration - though it requires better latency for any movements faster than basic hip and torso tracking.

I hope Microsoft does continue the product - it's a great sensor and has great capability. But the discrete GPU requirement is incompatible or onerous for many of its potential uses (entertainment (especially if that GPU is needed for graphics rendering) and IoT); and of the remaining use-cases, the latency is too slow for many in which the extra hardware costs are not a concern (robotics and industry).