Remove the requirement for a dedicated USB Host Controller

[AutomationIntegration]

My company's system requires multiple cameras, and the requirement of a USB Host Controller for each camera increases the hardware cost and the size of the computers involved. We would prefer to use ITX motherboards, but those have a single PCI-E slot which must be used for the GPU, as required for body tracking. That leaves us with mATX and ATX motherboards. Expansion cards with 4 host controllers are available, but in order to use 10 cameras that requires 3 such expansion cards (the motherboard's host controller + 2x 4-port cards gives only 9 host controllers) - this isn't possible on a mATX board because the GPU requires 2 of the 4 slots that the mATX form factor allows. So the system requires a full ATX motherboard and 3 USB expansion cards - with the oddity that the motherboard may have abundant USB ports of its own but only 1 of them will be usable for the cameras.

Our data logging shows that the cameras are nowhere near saturating USB 3.0 connections when used for body tracking, so there is more than enough bandwidth for multiple cards on a single host controller. Any additional latency from sharing a controller is likely tiny compared to the existing latency of the cameras.

Ideally, we would like to use 4 or more cameras in body tracking mode with a single USB host controller. This would make our builds and packaging significantly easier. It would allow us to skip expansion cards for a 4-camera system, and use only 1 expansion card for an 8-camera system.

[wes-b]

We don't have a requirement of a dedicated USB host controller per device. We do recommend it, but there is no hard requirement.

[AutomationIntegration]

@wes-b that is great to hear. But unfortunately, the Azure Kinect Viewer v1.3.0 reports otherwise, with one critical and two standard errors. I uploaded a screenshot to https://imgur.com/a/HR1MtQq

To reproduce: 1: Start first instance of Kinect Viewer. 2: observe that two devices are in the list. 3: hit Open Device when the first is selected. 4: open 2nd instance of Kinect Viewer 5: in the 2nd instance, press the Refresh Devices button. 6: observe the critical error and 2 plain errors in the second instance.

Expected result: No critical or standard error. A warning that multiple devices are on the same Host Controller would be useful, if applicable. A warning that some additional device(s) are unavailable would be satisfactory.

Actual result: 1 critical error "find_libusb_device(). libusb device(s) are all unavailable..." and two standard errors. Please note that it specifies all devices.

On the other hand, it will allow me to open and start both sensors simultaneously, and they will show the video and depth feeds correctly. It is not possible to conduct this test with the Body Tracking Viewer, as that only allows me to run a single instance.

I've not tried these tests with the very latest update to v1.4.0, as that is still labeled as alpha in the changelog.

[wes-b]

Gotcha, these error messages need to be cleaned up. They will not stop you though.

[MaxPalmer-UH]

@AutomationIntegration where did you go from here? Did you build a system and what was your experience?

[AutomationIntegration]

@MaxPalmer-UH We have the system running multiple sensors on each computer - the number per computer is limited by bottlenecks and requirement across the entire system, rather than just the port capacity. We can use two sensors on the same host controller (and probably more, but we don't need to test for more), but we only pull the depth data from the sensor - color data is more likely to hit the limits on the host controller. So, we have favored motherboards that have multiple host controllers and always try to give each sensor a dedicated host controller if it is practical, and otherwise limit the connection to two sensors per host controller. We don't need to exceed this because of the next factor.

Then USB cable length and quality is a separate issue that becomes relevant; the sensors will work with a high-quality 10 foot USB cable extension (again, we only use depth tracking) for a total length of 15 feet (10 ft extension + the 5 ft cable bundled with the sensor), and data degrades at any lengths greater than that. We have been using off-the-shelf extenders and could probably get better performance if we sought specialty high-quality ones, but it hasn't been needed. So, even if the computer could handle more sensors, our sensors are spread out and we need to manage the sensors as groups of 2 or 3 on each computer.

After that, the ability of the GPU to handle the depth data and body tracking becomes the limit. An RTX 2070 or 2080 can comfortably handle two sensors simultaneously, and the 2080 TI can handle 3. We may look towards the RTX 3070 or 3080 series, as those should handle 3 and possibly 4 sensors simultaneously. The RTX A6000 or RTX 3090 should be able to handle more, but we do not yet have any of the Nvidia Ampere cards to test with. We can put multiple cards into a computer, but at some point power consumption, heat management, and computer size become concerns. Hence, we limit ourselves to 3 per computer. If you are not using the body tracking SDK, you are not subject to these limits, and can add USB expander PCIe cards to handle as many sensors as you want until you hit some other bottleneck.

[MaxPalmer-UH]

@AutomationIntegration many thanks for taking the time to respond. The detail here is useful. I won't be doing body tracking so it looks like I could add more sensors to a single PC as long as I have USB controllers (via expansion cards) to match, Ultimately, I'm interested in the kind of point cloud merging and processing results you can get for volumetric capture of people with 5-7 Kinects. If anyone is doing this and is happy to share results, it would be great as I can only find much smaller numbers of sensors in online examples. I am also convinced that post processing could help remove the rough edges that appear at the boundary / edge of a person but which can be eliminated through having more views of the subject and some filtering,

[AutomationIntegration]

@MaxPalmer-UH of the work that isn't hidden behind proprietary systems, I think @Brekel's software is worth looking at. His Brekel Pointcloud v3 is very good, and sounds like exactly the software you need - it is at https://brekel.com/brekel-pointcloud-v3/ I can personally attest that it works with at least 5 sensors simultaneously (and probably more; I only tested with 5), and it can work across multiple computers on a local network if needed (very helpful if the sensors are spread further than the USB cables can stretch). It also has extremely useful automatic alignment tools.

[DrMaxP]

I've used Brekel's pointcloud V2 software, but need to retest the multi sensor auto alignment update in V3 as V2 wasn't reliable with my 7 Kinect V2 sensors. I believe it's been updated a fair bit since then. Will check it out again. If alignment works well then I'll be upgrading.

[christian-rauch]

As mentioned in https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1590#issuecomment-831930834 you can use at least two Kinects via the UVC interface as generic USB cameras on a single USB controller without issues.