Binocular Pupil Pro and USB bandwidth

[mkassner]

While testing and developing Pupil Pro binocular on different hardwares some important insights have be gathered. This only affects Hosts where Pupil binocular needs to share USB bandwidth with other devices.

Motivation

The new Pupil Pro binocular headset uses 3 cameras. 1 world camera (same as before) and 2 identical eye cameras. Since hardware with three separate USB controllers is super rare, I have chosen to integrate a usb hub controller in the collar clip. This clip was developed for Pupil Pro and allows operation with just on USB cable. A variation of the clip now bundles all three cameras on one usb controller.

General notes on USB bandwidth and image transport.

Usually UVC webcameras expose different image formats. The two common ones are YUV and MJPEG where the first is raw and the latter compressed. With the raw stream one can easily saturate the usb controller bandwidth and running image streams in parallel is limited to low res/ low framerate.

With the new pyv4l2 backend we always force mjpeg compression. This allow for more simulations streams. Decompression happens on the host and costs a bit more CPU time but we have that part quite optimized now. Running multiple compressed streams work remarkably well. We have successfully run dual 120fps vga streams (new pupil hi speed cameras) together with 720@30fps on one usb bus.

The devil is in the details

Setting uvc parameters trace=0x400, running pupil capture and then dumping the kernel log into a file dmesg > out.log, we can gain some insight into how UVC stream setup and bandwidth allocation works:

[ 3144.981603] uvcvideo: Device requested 309 B/frame bandwidth.
[ 3144.981614] uvcvideo: Selecting alternate setting 1 (1936 B/frame bandwidth).
[ 3144.981954] uvcvideo: Allocated 5 URB buffers of 32x1936 bytes each.
[ 3145.251282] uvcvideo: Device requested 309 B/frame bandwidth.
[ 3145.251292] uvcvideo: Selecting alternate setting 1 (1936 B/frame bandwidth).
[ 3145.251588] uvcvideo: Allocated 5 URB buffers of 32x1936 bytes each.
[ 3145.746490] uvcvideo: Device requested 732 B/frame bandwidth.
[ 3145.746501] uvcvideo: Selecting alternate setting 5 (800 B/frame bandwidth).
[ 3145.746534] xhci_hcd 0000:00:14.0: Not enough bandwidth. Proposed: 1787, Max: 1607
[ 3145.746540] xhci_hcd 0000:00:14.0: Not enough bandwidth

The amount of B/frame bandwidth is usually defined by the devise and uselessly conservative, usually prohibiting multiple streams per bus. This is where the bandwidth patch comes into play. It ignores the device bandwidth request and instead does an estimation based on frame size,rate and a compression parameter. The result is the value in the first line above (and 4th and 7th). The required bandwidth is not high: 309+309+732 B/Frame. This is very doable and will alow room for other devises on the bus like wifi,mouse and keyboard.

Unfortunately this is not the end of the story: Using the value the uvc driver now need to look at the USB device configuration Interface alternate settings and select the one that configures the isochronous endpoint such that it uses a few as possible small as possible transfers to achieve the required bandwidth. The device exposes different settings and one must be chosen. The Pupil world camera and the new Pupil hight speed cameras do this very nicely and offer altsettings in close intervals: See how the world camera offers the alternate setting 5 with 800B/Frame in line 8? The normal Pupil Pro eye camera is a bit different: it only needs 309 B/Frame but the smallest alternate setting offered is 1936 B/frame! Thus hogging bandwidth. This leads to the world camera being denied bandwidth by xhci in the last two lines. And from my understanding there is nothing we can do about this. Its a pupil pro eye camera firmware problem. What a waste.

When does this matter?

It does not matter when the USB is mostly free. Even with the oversized smallest alternate setting Pupil Pro bilateral works. This was confirmed on a Lenovo Thinkpad. But its tight: If you add another consumer (keyboard) via a hub on the same bus the last camera will not initialise with the error above. This is especially annoying when the device is internal like a wifi module (example: Surface Pro 1). UPDATE: read below. This problem is also related to USB3.0 controllers.

Using a preoccupied bus and Pupil Pro bilateral with normal eye cameras we are forced to reduce the world framerate to 15fps.

Since the Pupil high speed camera is well behaved in this regard, using it in a bilateral headset will be less prone to this problem despite its higher actual throughput! So here this does not happen and we can run 30fps for world and 120fps (depending on the bandwidth you have the option to reduce to 90fps) for each eye.

Summary

For binocular Pupil Pro we need two USB busses. The eye camera connection should be free of additional consumers. For binocular high speed Pupil Pro this is less of a problem. We can run three cameras on one bus.

[mkassner]

This seems to be related to XHCI vs EHCI bandwidth checking... manually setting the ports to EHCI

sudo setpci -H1 -d 8086:1e31 d8.l=0
sudo setpci -H1 -d 8086:1e31 d0.l=0

will change the error msg but does not solve the problem:

[  465.670700] uvcvideo: Device requested 309 B/frame bandwidth.
[  465.670711] uvcvideo: Selecting alternate setting 1 (1936 B/frame bandwidth).
[  465.671050] uvcvideo: Allocated 5 URB buffers of 32x1936 bytes each.
[  465.885594] uvcvideo: Device requested 309 B/frame bandwidth.
[  465.885604] uvcvideo: Selecting alternate setting 1 (1936 B/frame bandwidth).
[  465.886075] uvcvideo: Allocated 5 URB buffers of 32x1936 bytes each.
[  466.371589] uvcvideo: Device requested 732 B/frame bandwidth.
[  466.371599] uvcvideo: Selecting alternate setting 5 (800 B/frame bandwidth).
[  466.799808] uvcvideo: Allocated 5 URB buffers of 32x800 bytes each.
[  466.799835] uvcvideo: Failed to submit URB 0 (-28).

[mkassner]

Further investigation seems to indicate that this problem only happens on USB3.0 ports but not USB2.0.

On an IBM Thinkpad X201 we do not see any problems. A MacBook Air running Linux via SDCard as well as a Microsoft Surface Pro running Linux did have the issues described above.

This leads me to believe that while technically possible some USB 3.0 driver implementation details make running Pupil Pro bilateral with 30HZ cameras not possible.

Conclusion: In order to work across a variety of Host hardwares we need to satisfy the following constraints:

• Pupil Pro binocular needs a 2 USB port collar clip that groups the eye cameras.
• Pupil Pro binocular high speed can use a single usb clip that group all three cameras.

While there might be somebody out there with crazy USB/Kernel foo to make all versions of Pupil Hardware work with a single USB controller, it would always require more hacking and making sure it continues to work. We are flying very close to the sun.

[mkassner]

This same issue applies to the new libuvc based backend and thus to all thre OS'es.

[mkassner]

Recommendations for binocular Pupil Pro (30fps version):

Use lsusb to understand your USB device topology:

lsusb
Bus 001 Device 036: ID 17ef:480f Lenovo Integrated Webcam [R5U877]
Bus 001 Device 037: ID 17ef:480f Lenovo Integrated Webcam [R5U877]
Bus 001 Device 034: ID 05ac:8510 Apple, Inc. 
Bus 001 Device 033: ID 05ac:8404 Apple, Inc. 
Bus 001 Device 032: ID 046d:0843 Logitech, Inc. 
Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 002 Device 010: ID 0e0f:0008 VMware, Inc. 
Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The printout above indicates a problem: All three cameras are on Bus001. Make sure the world (c930e) is not on the same bus as the eye camera pair. To change the bus, all you need to to is plug the headset connections into a different usb port.

Furthermore: Try to avoid having other USB devices on the eye camera bus. The world camera does not mind sharing bandwidth with other devices.

[mkassner]

Our new cameras will support binocular high speed (and slower speed) recording in binocular configuration through a single USB bus. Beta hardware with slower camers has been fixed with dual bus USB clips.