Native linux backend based on usbfs

[badicsalex]

Hi,

TL;DR: Would you like to see a PR which uses usbfs ioctl's directly, even if it is not feature complete?

I've been using hidapi with good results for one of my projects. Unfortunately I have to port it to Android, and while I managed to do it with the libusb backend, but the build process is horrible, the licensing issues with libusb are worrying, and worst of all, the data comes in 5ms bursts (while it worked well on PC with the raw hidapi lib with almost exactly 1ms timings).

I suspect these bursts are caused by libusb and hidapi both making a thread (for a lot of back-and-forth sync-async conversions), and using mutexes and barriers to communicate, confusing the scheduler on the phone.

Right now I'm experimenting with using the USBDEVFS_BULK ioctl (which can actually also do interrupt transfers), and the code is turning out to be super simple. If it solves my jitter issues, I'll probably move to this solution for both android and linux. But it would be great to keep hidapi as it worked well in the past, and is also cross-platform.

I see that #27 has been open for a while, and this would be an even lower level replacement (albeit one that actually supports Android's restrictive permission model).

What do you think?

[ruabmbua]

To be clear, with usbfs you mean the interface that allows userspace direct control over USB devices? I did not know it had this name, usbfs usually means usb full speed?

Regarding #27 , its actually already implemented it, so I closed the issue.

If android really does not support hidraw, then I guess it would make sense to also make a implementation using sysfs + raw access to usb devices via usbfs.

Although it may make much more sense for you to create a small crate that handles stuff similar to rusb. You would only have to implement the bare basics to get hid working over it. Then we can use it in hidapi directly to implement hid stuff over the raw usb access crate.

[ruabmbua]

Even with its smaller scope, a separate crate implementing some wrappers over usbfs + sysfs for enumeration is probably a lot more helpful to other people if they want to use it.

[ruabmbua]

Also, since this will require enumerating directly via sysfs, maybe we can also put that into a separate crate, there is also some need by the rest of the ecosystem to have basic usb enumeration capabilities without using libudev c library. And android also does not have libudev.

I think it may be helpful to start a chat somewhere where I can invite other people who are probably interested.

I think if I remember correctly @Dirbaio from probe-rs side may be interested?

[badicsalex]

You were right the first time, usbdevfs (I forgot the dev part) is the files under /dev/bus/usb, libusb also uses these to implement its functionality on linux.

Now that I thought about it a bit more, I'll probably create a small crate (just like you said) that does control, interrupt and bulk requests on open file descriptors using pure rust on linux. It could then be further wrapped by hidapi (since hid read/write is usually just "regular" interrupt read/write). This would make hidapi much more convenient on android too.

Enumerating does sound like something that could be done in another separate crate (it's just a fancy readdir over /sys/bus/usb/devices anyway)

[badicsalex]

Looks like the library is going to be somewhat of a drop-in replacement for rusb.

There is one big limitation though: just like with rusb, there is no way to do actual non-blocking calls. The minimum delay is 1ms. This is different from what hidapi does, it uses a thread to always listen for incoming messages, and read reads from that; makng it possible to have non-blocking reads.

(Also it turns out that even with these low level kernel calls, I still get the packets in bursts, so the only reason to do this whole thing is to not having to compile libusb for android)

[whitequark]

Also it turns out that even with these low level kernel calls, I still get the packets in bursts

That's interesting--do I read it right that the interrupt endpoint specifies a 1000 Hz polling interval and you get data at 200 Hz?

[ruabmbua]

Looks like the library is going to be somewhat of a drop-in replacement for rusb.

There is one big limitation though: just like with rusb, there is no way to do actual non-blocking calls. The minimum delay is 1ms. This is different from what hidapi does, it uses a thread to always listen for incoming messages, and read reads from that; makng it possible to have non-blocking reads.

(Also it turns out that even with these low level kernel calls, I still get the packets in bursts, so the only reason to do this whole thing is to not having to compile libusb for android)

Is this a fundamental limitation of the usbdev uapi? Meaning if you submit something it will always yield to the OS for a minimum of 1ms?

[badicsalex]

do I read it right that the interrupt endpoint specifies a 1000 Hz polling interval and you get data at 200 Hz

I get most of it at 1000Hz, but occassionally (like, a few times a second), there are 5ms gaps. After the gap I get some in a row, but some are also lost. I haven't done that thorough of an investigation. I can if you're curious.

I just thought going with a lower level API would help, because I have seen strange latency issues with pthread mutexes before, but unfortunately it didn't help. Neither did adjusting thread priorities. I'll probably just implement a workaround for all this in the application layer.

Is this a fundamental limitation of the usbdev uapi?

Yes. The minimum timeout is 1ms, 0ms means forever. See: https://github.com/torvalds/linux/blob/b25f62ccb490680a8cee755ac4528909395e0711/drivers/usb/core/message.c#L339

Also in libusb: https://github.com/libusb/libusb/blob/8450cc93f6c8747a36a9ee246708bf650bb762a8/libusb/sync.c#L318

I think this is fundamental to how USB works: the device can only send to you if you have a pending URB for it, and that only happens once you entered the blocking call, and then you have to wait for at least the USB hardware poll interval thing.

BTW, this is the PoC code using usbdevfs directly from rust: https://github.com/badicsalex/tiny-linux-usb . It uses the blocking APIs right now. There is an URB API too; I plan to use instead of these easy but clunky blocking APIs, but that's more of a future plan.

[whitequark]

I think this is fundamental to how USB works: the device can only send to you if you have a pending URB for it, and that only happens once you entered the blocking call, and then you have to wait for at least the USB hardware poll interval thing.

Yes, this is correct.

I get most of it at 1000Hz, but occassionally (like, a few times a second), there are 5ms gaps. After the gap I get some in a row, but some are also lost. I haven't done that thorough of an investigation. I can if you're curious.

I am a little curious. HID reports at 1 kHz isn't a lot of traffic so it's likely a scheduling issue, but it's unclear where. As long as you have enough URBs queued the HCD should fill them at the bInterval rate autonomously.

How many URBs do you have queued concurrently? Could it be that for some reason or another, after an URB is received, your application isn't woken up in time to receive another one, and the HCD is not polling the device?

[badicsalex]

How many URBs do you have queued concurrently?

I was using hidapi with the libusb backend, so probably one.

I just made some logs to be sure:

Local Time    Time since    Difference between 
              last packet   contained timestamps (µs)

36:30.886  I  903.923µs     1000  
36:30.887  I  963.538µs     1000  
36:30.889  I  1.454461ms    1000  
36:30.894  I  4.765307ms    1000  
36:30.895  I  1.238923ms    1000  
36:30.898  W  3.380076ms    5000  
36:30.899  I  941.115µs     1000  
36:30.900  W  921.115µs     4000  
36:30.901  I  981.307µs     1000  
36:30.903  I  1.176462ms    1000  
36:30.903  I  722.884µs     1000  
36:30.904  I  992.577µs     1000  
36:30.905  I  906.231µs     1000  
36:30.906  I  999.154µs     1000  
36:30.907  I  864.962µs     1000  
36:30.908  I  1.058308ms    1000  

There are no actual bursts, but the device seems to have a buffer of at least 2: the 4ms gap only shows up 2 packets later in the stream. This confused me a bit. The 4ms wait is spent in the recv function both when using hidapi and when directly using the kernel API, so the problem is at kernel level or lower (could be the scheduler, but it can also be a faulty connector I think)

I'll try queuing more than one URB tomorrow and see if it's a scheduler issue and if I can capture the missing packets. If that doesn't help then I guess it's a problem with either the phone's USB driver or hardware, and packet loss is inevitable.