search

libusb / hidapi

A Simple cross-platform library for communicating with HID devices
https://libusb.info/hidapi/
Other
1.6k stars 391 forks source link

HID reports of more than 64 bytes and libusb backend #274

Open JoergAtGithub opened 3 years ago

JoergAtGithub commented 3 years ago

At Mixxx, users reported problems with USB 2.0 High-Speed devices, which are limited to Full-Speed mode for some reasons. Under MacOS and Windows they operate as High-Speed device.

This is not only a performance limitation, but it has functional implications for the HID communication, because USB Full-Speed is limited to a package size of 64 Bytes instead of 1024 in High-Speed mode. The result is, that HIDAP returns a 79Byte input report correct on Windows and MacOS, but two reports on Linux (libusb backend), the first with correct report ID, the second with report ID 0x00.

How could a cross platform application using HIDAPI handle this:

Linux-Output (libusb): Debug [Controller]: Traktor Kontrol S4 MK3 A79A_3: t:15558 ms, 64 bytes: 02 6E 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Debug [Controller]: Traktor Kontrol S4 MK3 A79A_3: t:15559 ms, 15 bytes: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This is the lsusb output: Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 3 bAlternateSetting 0 bNumEndpoints 2 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 0 bInterfaceProtocol 0 iInterface 11 Traktor Kontrol S4 MK3 HID HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 1.10 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 673 Report Descriptors: UNAVAILABLE Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x83 EP 3 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 1 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x02 EP 2 OUT bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 2 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 4 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 255 Vendor Specific Class bInterfaceSubClass 189 bInterfaceProtocol 0 iInterface 12 Traktor Kontrol S4 MK3 BD Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x03 EP 3 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 5 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 254 Application Specific Interface bInterfaceSubClass 1 Device Firmware Update bInterfaceProtocol 1 iInterface 10 Traktor Kontrol S4 MK3 DFU Device Firmware Upgrade Interface Descriptor: bLength 9 bDescriptorType 33 bmAttributes 7 Will Not Detach Manifestation Tolerant Upload Supported Download Supported wDetachTimeout 250 milliseconds wTransferSize 64 bytes bcdDFUVersion 1.10 Device Qualifier (for other device speed): bLength 10 bDescriptorType 6 bcdUSB 2.00 bDeviceClass 0 bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 bNumConfigurations 1

mcuee commented 3 years ago

I have asked the question in libusb-devel mailing list for confirmation. Hopefully experts there can check whether my answers are correct or not.

mcuee commented 3 years ago

Here is the reply from Tim Roberts, the expert in USB, and a top Windows Driver expert in the world.

For INPUT report (and also Feature IN report) the report is sent by the device.

On Tue, Jun 15, 2021 at 2:29 AM Tim Roberts wrote:

Xiaofan Chen wrote:

We have a question regarding the behavior of the Linux libusb (usbfs) behavior. https://github.com/libusb/hidapi/issues/274

This is not related to either libusb or usbfs.  This is all hidapi. Right?  Libusb doesn't know anything about the content of the data streams.

 From Ihor Dutchak:

I believe we can parse hid descriptor (which we already doing for some other purposes) and find out the largest report size, and use it as a length parameter passed to libusb_fill_interrupt_transfer.

Yes,  This, for example, is how the Windows HID subsystem operates.

Now some questions: 1) will libusb (or whatever backend there is) automatically concatenate several parts of the report? 2) what if we have a small report which is at least 2 times smaller than the largest report - is there a chance that two or more reports will be concatenated   into a single large buffer passed to transfer? My answers are below. 1) YES 2) NO

1) I'm not exactly sure what he's asking.  The parts of a complicated report are packet together by the device itself.  The report goes across the wire as a single unit.  No concatenation is necessary or possible. So, YES in that a single report is always sent as a single unit, but NO it's not done by the backend, it's done by the device.

2) NO.   Interrupt transfers are exactly like bulk transfers (Windows uses the same ioctl for both), in that a single short packet is going to complete the request.  No one is going to combine reports.  Remember that, assuming you have multiple reports, every report has a one-byte prefix giving the report ID. The rest of the transfer has to be just that report.

-- Tim Roberts Providenza & Boekelheide, Inc.

mcuee commented 3 years ago

Further clarification for OUTPUT report.

On Wed, Jun 16, 2021 at 12:30 AM Tim Roberts wrote:

Xiaofan Chen wrote:

For INPUT report yes the report is sent by the device. What about OUTPUT report? In that case, data will be sent by the host.

Yes, but a report is always a single unit.  That was the question, right?  One report == one transaction.  You always send the whole thing.  The HID library tries to make it seem like the individual fields are separate, but that's all just being composed in the library. Transmission to the device is always done one full report at a time.

-- Tim Roberts Providenza & Boekelheide, Inc.

mcuee commented 3 years ago

BTW, it is not that clear whether Windows has good support for high speed HID device. This is not related to this issue, just FYI only.

This is my question in OSR NTDEV forum and so far no answers. https://community.osr.com/discussion/292942/speed-of-high-speed-usb-hid-device

JoergAtGithub commented 1 year ago

Just for reference: USB HID spec 1.11 specifies this behaviour in chapter 8.4:

A report might span one or more USB transactions. For example, an application that has 10-byte reports will span at least two USB transactions in a low-speed device.

Youw commented 1 year ago

I believe we can parse hid descriptor (which we already doing for some other purposes) and find out the largest report size, and use it as a length parameter passed to libusb_fill_interrupt_transfer.

That's some piece of work.

mcuee commented 1 year ago

I believe we can parse hid descriptor (which we already doing for some other purposes) and find out the largest report size, and use it as a length parameter passed to libusb_fill_interrupt_transfer.

That's some piece of work.

The recent merge of #451 in git master should help, right?

Youw commented 1 year ago

I don't think #451 is actually needed for this particual one. Inside of the LIBUSB backend a Report Descriptor was always available (even though, the function for it was partially broken until #444).

But again: parse a descriptor, figure an each report size, have a "smart" read function, which will analyze the USB packet size, and concatenate the data when the report is larger than USB packet size - that is huge amount of work, and I don't see any volunteers.

mcuee commented 1 year ago

Advanced parsing and packet (re-)construction/verification was never the intention of hidapi. It always tried to be the thinest proxy library possible, with remarks to keeping the API uniform across platforms.

But again: parse a descriptor, figure an each report size, have a "smart" read function, which will analyze the USB packet size, and concatenate the data when the report is larger than USB packet size - that is huge amount of work, and I don't see any volunteers.

I think we can promote the hidraw backend for Linux and tell the user that libusb backend has various issues including this one (even go to the extreme to say libusb backend is only for legacy Linux versions which has issues with hidraw). In that case, this becomes a non-issue for Linux, macOS and Windows users.

Then the only users who are affected are FreeBSD users (hidapi is not really useful under NetBSD/OpenBSD because the user basically needs to rebuild the kernel to detach the kernel hid driver).

mcuee commented 1 year ago

@Youw

The other thing is to provide an advice for application developers using the libusb backend to work-around the issues. That may help as well.

mcuee commented 1 year ago

Then the only users who are affected are FreeBSD users

@cederom

Just wondering if you encounter such HID devices under FreeBSD. Thanks.

mcuee commented 1 year ago

Then the only users who are affected are FreeBSD users

@hselasky

Sorry to ping you as well. Do you think there is a way you can provide an insight whether FreeBSD libusb can provide some help here for hidapi users under FreeBSD. Thanks.

The reason I ask, is that I remember FreeBSD creates a ugen devce along with uhid so that HIDAPI can work under FreeBSD using the libusb backend. And FreeBSD libusb implementation is different from main libusb project.

cederom commented 1 year ago

Then the only users who are affected are FreeBSD users @cederom Just wondering if you encounter such HID devices under FreeBSD. Thanks.

Hey there :-) What / How can I test?

Are you sure this is not a problem of a cable / hub / bios settings / faulty device firmware?

For my USB HID mouse/keyboard all uses FullSpeed (12M):

ugen6.2: <Micro-Star INTL CO., LTD.> at usbus6, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen8.2: <Logitech USB Receiver> at usbus8, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (98mA)

Just a quick glance at different devel kits some of them choose HighSpeed (480M) some FullSpeed (12M) it depends on the attached device.. and from what I remember this is up to device to select speed it wants based on the electrical pull-ups-downs on the bus and on the protocol.

For my KT-LINK it looks like this:

# usbconfig -d 0.11 dump_device_desc
ugen0.11: <Kristech KT-LINK> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (100mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x0403
  idProduct = 0xbbe2
  bcdDevice = 0x0700
  iManufacturer = 0x0001  <Kristech>
  iProduct = 0x0002  <KT-LINK>
  iSerialNumber = 0x0003  <XYZ>
  bNumConfigurations = 0x0001

Some Infineon devkit:

# usbconfig -d 0.11 dump_device_desc
ugen0.11: <SEGGER J-Link> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x00ef  <Miscellaneous device>
  bDeviceSubClass = 0x0002
  bDeviceProtocol = 0x0001
  bMaxPacketSize0 = 0x0040
  idVendor = 0x1366
  idProduct = 0x0105
  bcdDevice = 0x0100
  iManufacturer = 0x0001  <SEGGER>
  iProduct = 0x0002  <J-Link>
  iSerialNumber = 0x0003  <XYZ>
  bNumConfigurations = 0x0001

ST-LINK V2:

# usbconfig -d 0.11 dump_device_desc
ugen0.11: <STMicroelectronics STM32 STLink> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x0483
  idProduct = 0x3748
  bcdDevice = 0x0100
  iManufacturer = 0x0001  <STMicroelectronics>
  iProduct = 0x0002  <STM32 STLink>
  iSerialNumber = 0x0003  <XYZ>
  bNumConfigurations = 0x0001

nRF52840PDK:

# usbconfig -d 0.11 dump_device_desc
ugen0.11: <SEGGER J-Link> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (100mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x00ef  <Miscellaneous device>
  bDeviceSubClass = 0x0002
  bDeviceProtocol = 0x0001
  bMaxPacketSize0 = 0x0040
  idVendor = 0x1366
  idProduct = 0x1015
  bcdDevice = 0x0100
  iManufacturer = 0x0001  <SEGGER>
  iProduct = 0x0002  <J-Link>
  iSerialNumber = 0x0003  <XYZ>
  bNumConfigurations = 0x0001
mcuee commented 1 year ago

@cederom

Ignore the USB speed part. Rather the issue is with USB HID device with HID report size greater than wMaxPacketSize of 64Bytes. Do you have such USB HID device?

cederom commented 1 year ago

@cederom Ignore the USB speed part. Rather the issue is with USB HID device with HID report size greater than wMaxPacketSize of 64Bytes. Do you have such USB HID device?

No clue sorry :-) For testing it may be good to create some sort of firmware on a cheap device like nRF52840-Dongle with USB Device block.. maybe BadUsb.. in order to be able to generate various situations on the controllable device firmware side?

hselasky commented 1 year ago

@mcuee: USB devices usually indicate end of data by sending a packet having size less than wMaxPacketSize.

There is a flag to allow short terminated packets. USB is packet oriented. When you setup a libusb transfer to receive 2x wMaxPacketSize, if any of those two packets are short, then that libusb transfer will complete.

The clear stall command may be used to sync/reset the stream, but many vendors don't care about implementing it.

As a general mechanism I would read wMaxPacketSize at a time. Then at each completion, you check the length of the content (this is given by the HID payload). If the payload is short terminated or you have enough data, then you proceed. This will have to be repeated every wMaxPaketSize bytes.

--HPS

hselasky commented 1 year ago

Of course, a device only sending wMaxPacketSize chunks of data may end up out of sync!

mcuee commented 1 year ago

From what I see, there is a work-around from the application side -- the application has to know the HID device and know the max HID report size.

I don't think, that it's neccessary to have wMaxPacketSize to implement this. The only information you need is, the size of each report by ReportID. And this information can be gathered from the report descriptor. With this information you could check if the report is complete, or if another data packet needs do be received.

That would require hidapi (or the application) to do some hacky guesswork. If the incoming packet is not equal to the size of a known report from the report descriptor, store it in a buffer. Then, when another incoming packet is another size that isn't in the size of a known report descriptor, concatenate it with the buffer.

There is no guessing work needed from hidapi or the application. The application developer has to know the device's each report size in the end.

mcuee commented 1 year ago

Advanced parsing and packet (re-)construction/verification was never the intention of hidapi. It always tried to be the thinest proxy library possible, with remarks to keeping the API uniform across platforms.

@Youw

I think we should maintain this way and not to make HIDAPI too complicated.

In this case, I think we can treat this issue as a documentation issue -- hidapi libusb backend will have such limitations and application will have to develop their own workaround if they encounter devices like this.

mcuee commented 1 year ago

@hselasky

Do you think that FreeBSD can develop its own hidapi compatible library and not use the hidapi libusb backend? License is not the issue here but because the libusb backend limitation.

hselasky commented 1 year ago

@mcuee : FreeBSD's libusb already has a backend abstraction:

https://github.com/freebsd/freebsd-src/blob/main/lib/libusb/libusb20_ugen20.c

What you could do is to enumerate all the FreeBSD HIDRAW devices and use some generic HID device configuration and endpoint profile, and emulate a USB device over other services in FreeBSD. Is that what you are asking?

--HPS

mcuee commented 1 year ago

@mcuee : FreeBSD's libusb already has a backend abstraction:

https://github.com/freebsd/freebsd-src/blob/main/lib/libusb/libusb20_ugen20.c

What you could do is to enumerate all the FreeBSD HIDRAW devices and use some generic HID device configuration and endpoint profile, and emulate a USB device over other services in FreeBSD. Is that what you are asking?

--HPS

Haha, what I am asking is a FreeBSD specific backend (eg: using hidraw) for HIDAPI so that FreeBSD HIDAPI users no longer need to use the libusb backend. The libusb backend for FreeBSD was a hack to get hidapi to work under FreeBSD as uhid can not be used.

I am not so sure if there are such hidraw driver for OpenBSD/NetBSD now.

Reference: https://reviews.freebsd.org/D27992

This driver provides raw access to HID devices through uhid(4)-compatible interface and is based on pre-8.x uhid(4) code.

Unlike uhid(4) it does not take devices in to monopoly ownership and allows parallel access from other drivers. hidraw supports Linux's hidraw-compatible interface as well.

mcuee commented 1 year ago

In the long term, I think the libusb backend will become not important at all and can be deprecated.

hselasky commented 1 year ago

Try asking: wulf@FreeBSD.org

mcuee commented 1 year ago

@wulf7

Just wondering if you are interested in this enhancement. It should benefit FreeBSD HIDAPI users.

wulf7 commented 1 year ago

Just wondering if you are interested in this enhancement. It should benefit FreeBSD HIDAPI users.

I already added hidraw enumeration support to https://github.com/wulf7/libudev-devd/ So, most probably, addition of linux hidapi backend to build is only thing left.

JoergAtGithub commented 1 year ago

Advanced parsing and packet (re-)construction/verification was never the intention of hidapi. It always tried to be the thinest proxy library possible, with remarks to keeping the API uniform across platforms.

And exactly this is not given here: It's not uniform accross platforms!

Using the libusb backend requires different code in the cross-platform applications build on top of hidapi. In case of Mixxx, we have mappings for over 100 devices, and it's very unlikely, that a mapping developer uses more than one operating system.

Different to the other backends, the libusb backend is not only a wrapper to a HID implementation - it's a HID implementation on it's own. Therefore it must contain more functionality than the other backends - and it does! For example it's allready the only backend, which contains an HID report descriptor parser - used to determine Usage and Usage-Table: https://github.com/libusb/hidapi/blob/b516e970480a8d05b1c392d106ca110a9d06d668/libusb/hid.c#L270-L273 It would be only a few lines to extend this existing parser to determine the size of each report.

mcuee commented 1 year ago

Using the libusb backend requires different code in the cross-platform applications build on top of hidapi. In case of Mixxx, we have mappings for over 100 devices, and it's very unlikely, that a mapping developer uses more than one operating system.

Just wondering why do you need to use libusb backend (other than for FreeBSD users)? I thought using hidraw was good enough under Linux?

mcuee commented 1 year ago

It would be only a few lines to extend this existing parser to determine the size of each report.

Hmm, are you saying it is trival to fix this paticualr issue? I get a different impression from the comment by Youw,

If that is the case, then PR will be welcome.

JoergAtGithub commented 1 year ago

Yes, we will switch our official Linux builds to hidraw backend in the next major release, due to this issue. (This will be a lot of hazzle for the end users due to changed udev rules.) But unofficial ports for BSD exist - and the maintainers of these ports do not have all these devices to debug the mappings on their platform.

I said it's easy to determine the size of all reports from the report descriptor. But just knowing the size, does not fix the data transport.

mcuee commented 1 year ago

But unofficial ports for BSD exist - and the maintainers of these ports do not have all these devices to debug the mappings on their platform.

Let's count on core FreeBSD developers like @wulf7 and @hselasky to fix the issue for FreeBSD users, by using the FreeBSD hidraw backend to replace libusb backend for hidapi.

cederom commented 1 year ago

@JoergAtGithub are these device mappings something that could be easily copied and recreated as static set of descriptors uploaded to a (physical/virtual) usb device that could emulate selected hid device or this require whole bunch of functioning devices of various kinds?

JoergAtGithub commented 1 year ago

Unfortunately not, each consist of several hundred to thousands lines of Javascript code. DJ controllers are fairly complex HID devices - this is why some have such large reports.

mcuee commented 1 year ago

@cederom

If you just want to test this issue, I think you can get an HID FW sample code for your preferred USB device, modify the FW to have wMaxPacketSize=64 bytes and the HID report size to be greater than 64Bytes, say 79 bytes to match the first report.

cederom commented 1 year ago

@JoergAtGithub I mean at the USB low level.. at the device descriptors.. hid descriptors and reports.. USB transfers.. I think of sniff-and-replay USB traffic for a given device that could be then replied as part of test automation.. would that make sense?

JoergAtGithub commented 1 year ago

Indeed, I missunderstood that. Test automation on USB low level wit a virtual device should be possible.

JoergAtGithub commented 1 year ago

But note that wMaxPacketSize must not always be defined by the HID device itself. The 64 bytes is the limit for USB full speed devices. But if you use a USB low speed hub to connect this USB high speed device, wMaxPacketSize is reduced to 8 bytes, and you will receive chunks of 8 bytes.

mcuee commented 1 year ago

But note that wMaxPacketSize must not always be defined by the HID device itself. The 64 bytes is the limit for USB high speed devices. But if you use a USB low speed hub to connect this USB high speed device, wMaxPacketSize is reduced to 8 bytes, and you will receive chunks of 8 bytes.

64 Bytes is not the limit for High Speed USB Interrupt Endpoint, but rather Full Speed USB Interrupt Endpoint.

However, it is not clear whether Windows even support high speed HID device or not. Please refer to my comment above.

BTW, it is not that clear whether Windows has good support for high speed HID device. This is not related to this issue, just FYI only.

This is my question in OSR NTDEV forum and so far no answers. https://community.osr.com/discussion/292942/speed-of-high-speed-usb-hid-devic

mcuee commented 1 year ago

USB 2.0 Spec https://www.usb.org/document-library/usb-20-specification

5.7.3 Interrupt Transfer Packet Size Constraints An endpoint for an interrupt pipe specifies the maximum size data payload that it will transmit or receive. The maximum allowable interrupt data payload size is 64 bytes or less for full-speed. High-speed endpoints are allowed maximum data payload sizes up to 1024 bytes. A high speed, high bandwidth endpoint specifies whether it requires two or three transactions per microframe. Low-speed devices are limited to eight bytes or less maximum data payload size. This maximum applies to the data payloads of the data packets; i.e., the size specified is for the data field of the packet as defined in Chapter 8, not including other protocol-required information. The USB does not require that data packets be exactly the maximum size; i.e., if a data packet is less than the maximum, it does not need to be padded to the maximum size.

From https://community.osr.com/discussion/292942/speed-of-high-speed-usb-hid-device

No, not at all. If (Windows support high speed HID device and) you have a maximum bandwidth endpoint, you'll get 3072 bytes per microframe. That means the 64kB transfer is going to be split up to span 21 microframes, or 3 complete frames. And that would be 24MB/s. If they don't support the multiple transactions, you'd get 1024 bytes per microframe, and the 64kB transfer would span 8 complete frames, at 8MB/s.

From https://learn.microsoft.com/en-us/windows-hardware/drivers/hid/hid-transports

Input/Output/Feature Report Length

The Hidclass/Hidparse driver pair defines lengths of HID Input, Output, and Feature Reports. The limit is 8 KB (minus 1 bit). Even if a HID minidriver can request a transfer of more than 8 KB for a report, only reports smaller than 8 KB are successfully transferred.

mcuee commented 1 year ago

But note that wMaxPacketSize must not always be defined by the HID device itself. The 64 bytes is the limit for USB high speed devices. But if you use a USB low speed hub to connect this USB high speed device, wMaxPacketSize is reduced to 8 bytes, and you will receive chunks of 8 bytes.

As mentioned above, you may want to change "high speed" to "full speed" in your comments. And it would be a good testing method for this issue if you can find a "USB low speed hub". At least I have not encountered one "USB low speed hub" myself since I was playing with USB device (from 2005 to now). I do have one USB full speed hub somewhere.

hselasky commented 1 year ago

But note that wMaxPacketSize must not always be defined by the HID device itself. The 64 bytes is the limit for USB full speed devices. But if you use a USB low speed hub to connect this USB high speed device, wMaxPacketSize is reduced to 8 bytes, and you will receive chunks of 8 bytes.

Yes, wMaxPacketSize may vary, but the principle about short terminated data transfers is the same. That's why it is important to send one byte extra, if the payload is exactly divisible by wMaxpacketSize. Else you risk getting out of sync at some point.

mcuee commented 1 year ago

Yes, wMaxPacketSize may vary, but the principle about short terminated data transfers is the same. That's why it is important to send one byte extra, if the payload is exactly divisible by wMaxpacketSize. Else you risk getting out of sync at some point.

HID spec says that there is an exception for the longest report.

https://www.usb.org/sites/default/files/hid1_11.pdf

8.4 Report Constraints The following constraints apply to reports and to the report handler:  An item field cannot span more than 4 bytes in a report. For example, a 32-bit item must start on a byte boundary to satisfy this condition.

 Only one report is allowed in a single USB transfer.

 A report might span one or more USB transactions. For example, an application that has 10-byte reports will span at least two USB transactions in a low-speed device.

 All reports except the longest which exceed wMaxPacketSize for the endpoint must terminate with a short packet. The longest report does not require a short packet terminator.

 Each top level collection must be an application collection and reports may not span more than one top level collection.

 If there are multiple reports in a top level collection then all reports, except the longest, must terminate with a short packet.

 A report is always byte-aligned. If required, reports are padded with bits (0) until the next byte boundary is reached.

hselasky commented 1 year ago

High-speed endpoints are allowed maximum data payload sizes up to 1024 bytes.

Some general comments:

1) Some embedded USB host controllers may have problems carrying such packets. These multi-DATA USB packet payload are sometimes incorrectly done on the wire. I have one USB device only working with EHCI from INTEL, and stops working on XHCI from INTEL. I sniffed the wire and there was a bug there how the PID's (packet type field) was encoded. I don't remember the exact details, but it was not according to spec, and some old USB HC's allowed that and so the product was shipped.

2) HID frequently use USB interrupt endpoints. One packet is exchanged per microframe (different between FULL/HIGH speed), and the data toggle must be the same, but doesn't have to toggle like BULK. Even if the hardware does it like this, there is something called interrupt moderation. USB jobs may be finished, but the USB host operating system doesn't see them until a CPU interrupt is generated. The number of IRQ's per second is 8000 by default for HIGH speed USB, if I'm not mistaken. Single jobs may not be submitted faster than this. Sometimes you can submit multiple jobs, but not all USB HC's can queue them in hardware, and still process them one by one. The next job is only started after the previous job has finished, and an IRQ has been generated.

mcuee commented 1 year ago

@hselasky

Do you know FreeBSD supports High Speed USB HID device with wMaxPacketSize of the Interrupt Endpoint = 1024 Bytes?

It is no clear whether Windows support it or not, so that usual HID devices only have wMaxPacketSize of the Interrupt Endpoint <= 64 Bytes.

hselasky commented 1 year ago

@mcuee : Yes, FreeBSD supports all features of the interrupt transport endpoint for all USB speeds.

FreeBSD doesn't support "USB stream mode endpoints".

Windows at least, did not support USB control transfers having payload above 4095 / 4096 bytes, last time I checked. FreeBSD can do up to 65535 bytes, by splitting the jobs, not needing to allocate a 64K buffer, which I guess the reason why Windows did not want to implement this.

Be-ing commented 1 year ago

If someone steps up to write an hidraw backend for FreeBSD (or modify the existing Linux one so it works with FreeBSD), I think deprecating the libusb backend, or at least documenting it as less favored, would be a decent way to handle this bug.

mcuee commented 1 year ago

Just want to mention that initial report is a bit confusing. Later it is clear that the speed is not the real issue (full speed or high speed or even Low Speed). The issue is the report size (one report will normally done in single USB transfer by the OS kernel HID driver, if not using libusb) can be greater than packet size.

[To be confimred -- confirmed to be wrong] It seems to me because Windows inbox HID driver most likely does not really support Interrupt IN/OUT Endpoint with wMaxPacketSize greater than 64 bytes, so for the High Speed USB HID device out in the market, the Interrupt IN/OUT Endpoint will still have wMaxPacketSize at 64Bytes.

Later version of Windows may or may not have such limitations but the device out in the market have to support older version of Windows. [/To be confirmed -- confirmed to be wrong]

On the other hand, the HID report size can be bigger. Again Windows is the limitation here at 8KB.

Same for low speed USB HID device where the Interrupt Endpoint will have wMaxPacketSize at most 8 Bytes. Again the report size can be bigger

In any case, the HID report size (IN, OUT, Feature report) can be greater than wMaxPacketSize of the respective Interrupt Endpoint or Control Endpoint. Therefore hidapi libusb backend has to know the report size to decide whether it has finished processing of the report or not.

If we ask the application developer to carry out workaround in the application, it may be a big issue for the more complex applications like Mixx in this issue report (with mappings for over 100 devices).

mcuee commented 1 year ago

Just an example of High Speed USB HID device -- Microchip PICKit 4 (USB Composite Device, with first interface to the HID interface). You can see that the wMaxPacketSize of the Interrupt IN endpoint for the HID interface is still 64 bytes.

But in this case, the report size is also 64 Bytes so it will not encounter issues using hidapi-libusb.

lsusb -t info under Ubuntu Linux 20.04.

mcuee@UbuntuSwift3:/sys/bus/usb/drivers/usbhid$ lsusb -t
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 10, If 0, Class=Human Interface Device, Driver=usbhid, 480M
        |__ Port 1: Dev 10, If 1, Class=Communications, Driver=cdc_acm, 480M
        |__ Port 1: Dev 10, If 2, Class=CDC Data, Driver=cdc_acm, 480M

USBTreeviewer connection information under Windows 11.

Connection Index         : 0x01 (Port 1)
Connection Status        : 0x01 (DeviceConnected)
Current Config Value     : 0x01 (Configuration 1)
Device Address           : 0x08 (8)
Is Hub                   : 0x00 (no)
Device Bus Speed         : 0x02 (High-Speed)
Number Of Open Pipes     : 0x05 (5 pipes to data endpoints)
Pipe[0]                  : EndpointID=1  Direction=IN   ScheduleOffset=0  Type=Interrupt  wMaxPacketSize=64   bInterval=1
Pipe[1]                  : EndpointID=2  Direction=OUT  ScheduleOffset=0  Type=Interrupt  wMaxPacketSize=64   bInterval=1
Pipe[2]                  : EndpointID=4  Direction=IN   ScheduleOffset=0  Type=Interrupt  wMaxPacketSize=64   bInterval=16
Pipe[3]                  : EndpointID=3  Direction=IN   ScheduleOffset=0  Type=Bulk       wMaxPacketSize=512  bInterval=0
Pipe[4]                  : EndpointID=5  Direction=OUT  ScheduleOffset=0  Type=Bulk       wMaxPacketSize=512  bInterval=0

hidtest output under Linux.

Device Found
  type: 03eb 2177
  path: /dev/hidraw4
  serial_number: BUR200973052
  Manufacturer: Microchip Technology Incorporated
  Product:      MPLAB PICkit 4 CMSIS-DAP
  Release:      100
  Interface:    0
  Usage (page): 0x1 (0xff00)
  Bus type: 1 (USB)

  Report Descriptor: (35 bytes)
0x06, 0x00, 0xff, 0x09, 0x01, 0xa1, 0x01, 0x15, 0x00, 0x26, 
0xff, 0x00, 0x75, 0x08, 0x96, 0x40, 0x00, 0x09, 0x01, 0x81, 
0x02, 0x96, 0x40, 0x00, 0x09, 0x01, 0x91, 0x02, 0x95, 0x04, 
0x09, 0x01, 0xb1, 0x02, 0xc0,

HID report patser output

0x06, 0x00, 0xFF,  // Usage Page (Vendor Defined 0xFF00)
0x09, 0x01,        // Usage (0x01)
0xA1, 0x01,        // Collection (Application)
0x15, 0x00,        //   Logical Minimum (0)
0x26, 0xFF, 0x00,  //   Logical Maximum (255)
0x75, 0x08,        //   Report Size (8)
0x96, 0x40, 0x00,  //   Report Count (64)
0x09, 0x01,        //   Usage (0x01)
0x81, 0x02,        //   Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
0x96, 0x40, 0x00,  //   Report Count (64)
0x09, 0x01,        //   Usage (0x01)
0x91, 0x02,        //   Output (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
0x95, 0x04,        //   Report Count (4)
0x09, 0x01,        //   Usage (0x01)
0xB1, 0x02,        //   Feature (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
0xC0,              // End Collection

// 35 bytes
mcuee commented 1 year ago

Past discussions and comments by the original author of HIDAPI Alan Ott.

On 30-Jan-2013

It's more complicated than that, and it needs to be done on the device side. You can have HID reports (interrupt transfers) longer than 64-bytes, but on full-speed connections you will need to send them as 64-byte transactions, the same as you'd do for a long bulk transaction (with a zero-length-packet at the end, if needed, per the USB spec).

It will take more than one frame to do this though. Interrupt endpoints only get one IN token per frame (and on Windows, it seems they get one every other frame). The HID code in windows, mac, and linux/hidraw will piece them together. The libusb HIDAPI implementation will not and you'll get each piece separately. This is a longstanding limitation (which can be worked around), which I'll get to one day (patches appreciated).

mcuee commented 1 year ago

Past discussions and comments by the original author of HIDAPI Alan Ott.

On 25-Feb-2014

It's a rather big problem if you are affected by it. In reality, almost no devices have reports larger than 64 bytes. VRPN simply works around it.

The real solution is to parse the report descriptor and determine the maximum report length and use that length as the maximum transfer size. HIDAPI currently does not parse report descriptors.

Setting the length to 512, or some arbitrarily large number, will cause libusb to buffer transactions until it gets to that number of bytes (or until it receives a short transaction), and then return all the data as a single transfer. This is probably not what you want.

If you have a report which is 80 bytes long, for example, set the length to 80. Libusb will then collect transactions until 80 bytes are reached and return it as one transfer.

This whole issue has to do with multi-transaction transfers. There's nothing in the underlying USB protocol which says that a transfer has ended, or whether the transfer has more data. Yes, a short transaction does indicate end-of-transfer, but an endpoint-length-sized packet does not necessarily indicate that the transfer will contain another transaction. It's up to the agreed-upon protocol between the device and host. For HID devices, the agreed-upon protocol is described by the report descriptor, which HIDAPI does not have a parser for.