Question regarding USB and the Windows API

[kdschlosser]

OK let me give you some information on what I am trying to do and what I am using usbpcap for, and where I am having some difficulty because of my lack of how USB works in Windows.

Windows has an API for using DeviceIoControl to interact with eHome CIR devices (MCE IR Receivers). This API is not documented very well, it is not complete and it also contains incorrect information... I am writing a Python binding to receive and decode CIR signals from Windows... and also transmit (blast) IR as well.. I keep on getting random application crashes when the program runs.. I am not 100% sure as to why and I am not able to locate the source of the problem.. When I debug the thing it points to a portion of the Python core that handles garbage collection... I have written a lot of programs that interact with the Windows API so I am not a newbie at it.. I am still learning tho.. There can be all kind of odd behavior if things are not done correctly, with the lack of accurate and complete documentation from Microsoft being able to write the python binding is turning out to be a huge amount of guess work.

What I decided to do instead was I was going to go after the low level data that is sent along the USB.. I am using usbpcap to capture the data so I can write the code to parse it... which I have done.... Now I need to be able to read the data..

Reading the data is turning out to be another problem... Lack of complete documentation is the largest problem..

I looked at using the WinUSB portion of the API.. as it turns out this is not going to work.. In order to use that portion of the api one of the driver files needs to be winusb.sys... I then thought about using the HID portion of the API.. the ehome drivers are HID, they carry the hid.dll as part of the driver package.. This does not work either...

I am able to obtain a file handle to the device. and in the 2 above mentioned API's that would be used to get the endpoint handle.. and the endpoint handle would then be used to read\write to the pipe. When I try to acquire the the endpoint handle is where I am stuck., those 2 API's will not work with the device.

HOw would I go about reading and writing to the device? The information on DeviceIoControl and all of the control codes is not complete.

Does anyone have any suggestions on how to go about doing this?

[desowin]

If the device is HID, then you should be able to determine the report format by parsing the HID descriptor. The CONFIGURATION descriptor would be useful in order to determine if the device uses only HID reports or also has some additional interfaces. You can simply capture the enumeration (start capture before connecting device and then connect the device) and analyze it in Wireshark.

If the device uses WinUSB, then you can use libusb. When using libusb, the same code can work on Windows and Linux. The pyusb can be used to use libusb in Python. Example project using pyusb together with libusb and WinUSB for the target device is pystlink.

[kdschlosser]

The funny thing is, the device is an HID device but.... I am not able to use any of the Hid* windows functions on the device... When the device gets installed it also installs several other drivers as well. some of these drivers do have a pid and vid and some do not. some contaoin a path as a hardware id, the ones that do contain a vid and a pid do not match the pid and vid of the actual manufacturer they are something else. I have not yet looked into it.. These drivers/devices are part of the USDB driver class.. however they cannot be accessed by using either the Hid or the WinUSB windows functions...

I have tried using pyusb which uses the libusb dll. this works only for the driver/device that has the pid and vid of the manufacturer. No other driver/device shows even when enumeration all drivers/devices using libusb.

This is where I am stuck.. I am able to get a file handle to the "ir device" using the IR device setup class GUID. this however does not point to the piece of physical hardware. instead it points to one of the "virtual" ones. in order to target the specific device using libusb I need to have the vid and the pid for the physical device.. I am trying to find a way to connect the virtual driver to the physical one. as of yet I have not found anything that connects the 2 together.. because there are a plethora of manufacturers for MCE remote IR receivers it would be a pretty large lookup table of vids and pids to have to reference against. doing that would be the wrong way to go about it anyway..

There has to be some way of marrying the virtual driver to the physical one in order to obtain the vid and pid so it can be used in libusb.. once I am able to do that sending and receiving raw data should be simple to do.

I have all of these "pieces" but no way to associate them to each other...

[kdschlosser]

This is only going to be "Windows ONLY" for now.. I do plan to make it so that these "Windows ONLY" devices will be able to be used in Linux and OSX. I have already written all of the code that parses the low level data for an incoming IR signal and I have written code that decodes some 30 different IR protocols.. I am still adding to that..

Microsoft did do something right for once.. Whatever employee at Microsoft that wrote it needs to be given a BIG thank you. MCE remote controls use a slightly modified version of either the RC5 or the RC6 protocol.. do not remember which one off hand. it doesn't matter tho because the difference between those 2 protocols is simply the number of bits being sent and the arrangement of the bits.. Those protocols operate around the 36kHz frequency.. the specification was written so that the receivers needed to be able to read IR code anywhere between 30Khz and 60kHz. and the specification was written to handle pulse timings as low as 50us and the receiving hardware is responsible for reading/sending the IR signals. so there is no problem with receiving IR or sending IR where the timings on the code can get all messed up due to processor time.. the receiver has it's own processor for doing this... so all that needs to get sent to the receiver is the duration to flash the led for and the duration to not flash the led for.... the receiver/blaster handles modulating the "on" for the duration that is given. so for a frequency of 38kHz the ir would need to change state every 26.3157894736842105263157894736842105us (microseconds) or 0.0000263157894736842105263157894736842105 seconds.. Now computers are indeed fast.. but handling this while also having to process GUI's and all of the rest of the goodies a PC processes can only be done a single way.. and that is to give the program that is handling this a higher CPU priority then the OS. Because of how the specification is written it offloads that task to the receiver, thus eliminating that "snafu" and because it was written so that the device has to be sensitive to such a wide frequency range and have a sensitivity of 50us for the reported burst timings this makes it a pretty decent low cost "plug and pray" means to receive IR and to transmit IR fairly accurately. They made it like this because of the ability to learn and blast IR. so it had to be designed to cover the majority of the CIR devices available today... It can also be used for 2 way IR communications as seen with a lot of portable/mini-split air conditioners available today.

There is no reason why these devices should not be able to be used on other operating systems like as raspberry pi. Because Microsoft loves to control everything. the driver for the device is not a manufacturer supplied driver. it is a Microsoft driver. So in order for the device to work with the driver the data sent over the USB is going to be identical from manufacturer to manufacturer. I have the specification for what the format of the data is.. but because this "device type" falls into the vendor specific category I need a way to determine what the device is if enumerating all USB devices. or to get the vid and the pid using Windows (for the time being)..

That being said.. I know 0 about USB.. But I am a fast learner.. only started messing about with this IR stuff a week and a half ago. I am sure with many more hours of learning and the "try/fail" routine I will be able to locate what is needed.. I am hoping to save myself from having to do that with some ideas thoughts and insights.

I have attached the specification.. the software/data portion of the specification starts at page 59. the USB portion starts at page 72ish I believe.. before page 59 is the physical specifications

196064_mce_ehome_device_specification03-08-2011-V2_1.pdf

[kdschlosser]

bah! another road block..

I do not know if usbpcap handles Microsoft OS Descriptors for USB. This is the descriptor that is used to determine the "device type" and to load the proper driver for the device.

https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/microsoft-defined-usb-descriptors

I do not see any message for this using wireshark. I have uninstalled the device. started monitoring the port and then plugged the device in. Windows went through the whole device installation process and this never came up.

there is a whole lot of information about a device that is in that descriptor that is windows specific. This is something I would need to query the device for in order to know if the device is an IR receiver.. That piece of information I need in order to do what I am trying to do is contained within this descriptor.

It would be nice if usbpcap was able to display packet data for this descriptor.

[desowin]

The funny thing is, the device is an HID device but.... I am not able to use any of the Hid* windows functions on the device... When the device gets installed it also installs several other drivers as well. some of these drivers do have a pid and vid and some do not. some contaoin a path as a hardware id, the ones that do contain a vid and a pid do not match the pid and vid of the actual manufacturer they are something else. I have not yet looked into it.. These drivers/devices are part of the USDB driver class.. however they cannot be accessed by using either the Hid or the WinUSB windows functions...

This sounds like composite device. Could you please attach pcap file contaiting the enumeration?

I have tried using pyusb which uses the libusb dll. this works only for the driver/device that has the pid and vid of the manufacturer. No other driver/device shows even when enumeration all drivers/devices using libusb.

It only works for the drivers that are compatible with libusb (WinUSB, libusb-win32, libusbK). If a function driver takes over interface it won't show in libusb.

Every USB device has VID/PID. USB device can implement multiple functions (composite device). Each of the function is seen separately in Device Manager.

There has to be some way of marrying the virtual driver to the physical one in order to obtain the vid and pid so it can be used in libusb.. once I am able to do that sending and receiving raw data should be simple to do.

If USBPcapCMD lists the "IR entry" in the devices list, then you should be able to use CM_Get_Parent and other CfgMgr32.dll functions to "marry" them.

It would be nice if usbpcap was able to display packet data for this descriptor.

Remove the HLKM\SYSTEM\CurrentControlSet\Control\UsbFlags\vvvvpppprrrrr key where vvvv is VID, pppp is PID and then repeat.

[desowin]

It would be nice if usbpcap was able to display packet data for this descriptor.

Remove the HLKM\SYSTEM\CurrentControlSet\Control\UsbFlags\vvvvpppprrrrr key where vvvv is VID, pppp is PID and then repeat.

I just realised that this won't really work. You'll basically get packet that Wireshark 3.2.0 will mark with expert info that USBPcap did not recognize URB_FUNCTION_GET_MS_FEATURE_DESCRIPTOR (#78 is the issue to track it).

If you want to see the actual data transferred on the wire in near future, you could get OpenVizsla hardware sniffer, then delete the key from registry, start capture (e.g. using ovextcap from matwey/libopenvizsla) and connect device. If doing so, you might want to enable NAK and SOF filtering to make the capture contain only relevant data.

[kdschlosser]

I guess I am confused as to why the URB_FUNCTION_GET_MS_FEATURE_DESCRIPTOR cannot be used.

OK so this i guess is where I want to ask some questions regarding usbpcap.. in wireshark there isn't the ability to ask for various information. it is a logging routine. so it is going to log what the USB is currently doing..

If I wanted to write a python binding and attach a GUI to it using wxPython is it possible to ask for descriptors on demand using usbpcap???

I know that a direct binding could not be made and I would probably have to either use cython to write the binding or I would have to write an extension module to interface with it either way doesn't matter because I am able to do both. There is also the executable that is included with usbpcap. I do not know exactly what this is for or how to use it. I might tinker a bit and try and add URB_FUNCTION_GET_MS_FEATURE_DESCRIPTOR to the code. I need to figure out why in the hell i cannot get wdk installed into visual studio.

[desowin]

I guess I am confused as to why the URB_FUNCTION_GET_MS_FEATURE_DESCRIPTOR cannot be used.

Because it is not implemented. The problem is the need to translate the data from _URB_OS_FEATURE_DESCRIPTOR_REQUEST into SETUP token (8 bytes) in a way that matches what is sent on the wire. There have been issues with that in the past (#83). I would like atleast some statistical evidence (in other words traces done by USBPcap with the feature implemented and hardware sniffer) to give a green light on this.

The goal behind USBPcap was to make it useful for generic USB analysis, not only for Windows driver debugging. That is, someone who is aware of how USB works (e.g. someone who wants to write Linux driver for some USB device) can capture how the official Windows drivers talks to the device, without the need to gain deep understanding of Windows OS-specific mechanisms.

OK so this i guess is where I want to ask some questions regarding usbpcap.. in wireshark there isn't the ability to ask for various information. it is a logging routine. so it is going to log what the USB is currently doing..

Yes, USBPcap is just a logger.

If I wanted to write a python binding and attach a GUI to it using wxPython is it possible to ask for descriptors on demand using usbpcap???

No, but you could do what USBPcapCMD is doing (when inject descriptors option is active) and fake it in user-space.

I know that a direct binding could not be made and I would probably have to either use cython to write the binding or I would have to write an extension module to interface with it either way doesn't matter because I am able to do both.

Just open the \\.\USBPcapX where X is integer representing a root hub and issue IOCTL does defined in USBPcapDriver/include/USBPcap.h. The capture data can be read using ReadFile(). Note that USBPcap requires elevated priviledges to read the data (because I want the user to be aware when some application tries to read USB traffic - e.g. keyboard inputs).

There is also the executable that is included with usbpcap. I do not know exactly what this is for or how to use it.

It is essentially a Wireshark extcap with the option to use it as standalone commandline tool. It makes Wireshark know how to access the \\.\USBPcapX without requiring any USBPcapDriver specific code in Wireshark.

[kdschlosser]

Well at this point I am again stuck. It would appear as tho I am not able to read any data using pyusb (libusb-1.0) It kicks out an error stating that is not supported on my platform... Kind of makes the library useless for the most part if it is being run on Windows.

I am wondering if I am able to read the data using DeviceIoControl.

There are 2 reasons why I am trying to do this. first off is issues with that whole garbage collection app crash that is happening.. The other is because all of the CIR api in windows has been deprecated.

Microsoft removed Media Center in windows 10. They didn't remove the ehome portions. I believe they may have either forgotten or wanted to deprecate this portion of it.. either or it doesn't really matter. It means that at some point it is going to get removed. that is going to leave these devices in a state where they cannot be used.. even if the drivers still work. without an API to access the thing no data can be had unless there is a way to read/write the low level data.

The biggest issue with these devices is they are Windows only devices. they make use of the before mentioned descriptor which is a Windows only descriptor. In my use case a developer would have to have a working knowledge of how windows handles USB in order to write a driver for this device on any other platform. buried in that descriptor is some kind of an identifier that is used to determine that the device is a CIR device. and this is how Windows installs all of the other drivers to be able to use it. another downside is I am not able to find a link between the driver that is used for the CIR api and the physical device. not even in the registry.. I still have not tackled that issue. seeing as how I only have a single device I am able to enumerate the generic USB devices and pull the one that is using the cir setup class.

the only way a driver could be developed for this device to run on anything other then windows without having any windows knowledge is there would have to be a list of known CIR device vids and pids.

[desowin]

Well at this point I am again stuck. It would appear as tho I am not able to read any data using pyusb (libusb-1.0) It kicks out an error stating that is not supported on my platform... Kind of makes the library useless for the most part if it is being run on Windows.

If you understand the USB transfers well enough, then you can just use Zadig to install libusb supported driver for the device (e.g. libusbK). Then libusb will work with it. However, you'll lose the access to CIR API.

Microsoft removed Media Center in windows 10. They didn't remove the ehome portions. I believe they may have either forgotten or wanted to deprecate this portion of it.. either or it doesn't really matter. It means that at some point it is going to get removed. that is going to leave these devices in a state where they cannot be used.. even if the drivers still work. without an API to access the thing no data can be had unless there is a way to read/write the low level data.

Just like above, Zadig can help if it gets removed. But then your driver should include everything that is needed to get it working (and at this point, it should just work fine on Linux).

The biggest issue with these devices is they are Windows only devices. they make use of the before mentioned descriptor which is a Windows only descriptor.

This OS descriptor (0xEE string descriptor) is solely so Windows can pick up driver from a Microsoft classes list. MTP also has the OS descriptor and there are Linux applications to use MTP (e.g. Android File Transfer).

the only way a driver could be developed for this device to run on anything other then windows without having any windows knowledge is there would have to be a list of known CIR device vids and pids.

Get a list, or just ask the potential device for string descriptor 0xEE and parse that... Alternatively you might check what lies in the CONFIGURATION descriptor which is, in my opinion, far more important.

[kdschlosser]

The device has a type of generic USB device. to know that it is a CIR device this has to be done using the OS descriptor. I did manage to get the CIRAPI working properly There is a bug in python i seem to have stumbled upon. it is not a documented bug either. I did manage to work around the problem. I am happy with that. I still want to get these things so they can work on another platform..