vpelletier / python-functionfs

Pythonic API for linux's functionfs
GNU General Public License v3.0
40 stars 13 forks source link

how to add MS OS descriptors? #9

Closed monkey-jsun closed 6 years ago

monkey-jsun commented 6 years ago

Hi, Vincent, et al,

It appears python-functionfs actually support MS OS descriptor. Is there a sample code to show how it works?

Specifically I like to set compatibility ID to "WINUSB". From the MS WCID doc, it appears I would also need to set an extended property, "DeviceInterfaceGUID". Is it pretty much it?

Appreciate as always!

vpelletier commented 6 years ago

First, I must say that I did not use it directly, so take the code with a grain of salt - bugfixes very welcome. I implemented its support following kernel API and transposing how I implemented regular descriptors.

Intended use is:

And as for how these descriptors actually work internally/how windows interprets them, I remember finding an MSDN page when I wrote that code, but I did not pay much attention beyond getting the general feeling of what these descriptors are needed for.

monkey-jsun commented 6 years ago

Hi, Vicent,

I tried to follow your directions and am having some difficulties.

It appears I would need to setup the following

I'm not how these understanding map into what you said:

Thanks a lot in advance.

vpelletier commented 6 years ago

I would expect MSDN do have answers all these questions, as these descriptors were designed by microsoft and certainly documented them so USB device makers could populate these properly.

vpelletier commented 6 years ago

Actually, this is wrong for the fixed-index descriptor string. This is a linux implementation detail, which is supported, but separately from OS descriptor passed to functionfs. This looks like it belongs to configfs:

What:           /config/usb-gadget/gadget/os_desc
Date:           May 2014
KernelVersion:  3.16
Description:
                This group contains "OS String" extension handling attributes.

                use             - flag turning "OS Desctiptors" support on/off
                b_vendor_code   - one-byte value used for custom per-device and
                                per-interface requests
                qw_sign         - an identifier to be reported as "OS String"
                                proper
monkey-jsun commented 6 years ago

Hi, Vincent,

Yes, I can confirm that bMS_VendorCode and qw_sign are set in configfs as you suggested.

And functionfs only need to set the following 2 properties:

Does the following code looks rougly right to you? (Not tested yet, just a sanity check)

        ext_list = []
        ext_list.append(functionfs.getOSExtPropDesc(1, unicode("DeviceInterfaceGUID"), unicode("{8FE6D4D7-49DD-41E7-9486-49AFC6BFE475}")))
        ext_list.append(functionfs.OSExtCompatDesc(bFirstInterfaceNumber=0, CompatibleID="WINUSB"))
        ...
        os_list = []
        os_list.append(functionfs.getOSDesc(0, ext_list))
        ...
        super(FunctionFSTestDevice, self).__init__(
            path,
            fs_list=fs_list,
            hs_list=hs_list,
            os_list=os_list,
            lang_dict={
                0x0409: [
                    INTERFACE_NAME,
                ],
            },
        )
monkey-jsun commented 6 years ago

If things don't work (most likely), what is the best way to debug?

Is there a way to dump the serialized descriptor byte stream?

monkey-jsun commented 6 years ago

Hi, Vincent,

I got pretty far along. See the following code. Now I have one error that I cannot figure out.

    ...
    ext_list = []
    compatDesc = functionfs.OSExtCompatDesc()
    compatDesc.bFirstInterfaceNumber = 0
    # CompatibleID : U8[8]
    compatDesc.CompatibleID=(ctypes.c_ubyte * 8).from_buffer_copy(b"WINUSB\0\0")
    ext_list.append(compatDesc)
    # note ending \0 is required by MS spec
    propDesc = functionfs.getOSExtPropDesc(1, "DeviceInterfaceGUID\0".encode("utf-16-le"), "{8FE6D4D7-49DD-41E7-9486-49AFC6BFE475}\0".encode("utf-16-le"))
    ext_list.append(propDesc)
    os_list = []
    os_list.append(functionfs.getOSDesc(0, ext_list))
    try:
        super(FunctionFSTestDevice, self).__init__(
            path,
            fs_list=fs_list,
            hs_list=hs_list,
            os_list=os_list,
            lang_dict={
                0x0409: [
                    INTERFACE_NAME,
                ],
            },
        )

Below is the error I got during running. It does not seem to like type we have in ext_list. For some reason, I cannot print the ext_type variable. As such, I cannot really trace through the code to understand why the error is thrown.

Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 220, in main(*sys.argv[1:]) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 192, in main with FunctionFSTestDevice(path) as function: File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 105, in init os_list.append(functionfs.getOSDesc(0, ext_list)) File "/home/pi/usb-gadget/python-functionfs/functionfs/init.py", line 128, in getOSDesc raise TypeError('Extensions of unexpected type') TypeError: Extensions of unexpected type Exception ignored in: <bound method Function.del of <main.FunctionFSTestDevice object at 0xb69d0d30>>

vpelletier commented 6 years ago

About how to dump the descriptor bytestream, my first try would be to install wireshark on the windows host and setup usb sniffing. There is usbmon on linux side, but I do not remember testing it on gadget-side kernel, only as a host - hopefully it will work just the same.

Below is the error I got during running.

Woops, I'm using isinstance between a class and tested class. This should be fixed in 589ec57a9a6aebab7134f83a58752d56dac0670c .

It does not seem to like type we have in ext_list. For some reason, I cannot print the ext_type variable. As such, I cannot really trace through the code to understand why the error is thrown.

This is not expected. This value should be a ctype structure (sub)class (in your case, the OSExtCompatDesc class itself). Maybe output does not get flushed before the error happens and interpreter exits ? (but even though, I would expect python to flush it...)

monkey-jsun commented 6 years ago

Thanks, Vincent.

It appears we need another typo fix. See below. Right?

@@ -106,10 +108,12 @@ def getOSDesc(interface, ext_list): List of instances of extended descriptors. """ try:


After this fix, I now get another error which I'm not sure about.

<class 'functionfs.functionfs.OSExtCompatDesc'> Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 220, in main(sys.argv[1:]) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 192, in main with FunctionFSTestDevice(path) as function: File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 105, in init os_list.append(functionfs.getOSDesc(0, ext_list)) File "/home/pi/usb-gadget/python-functionfs/functionfs/init.py", line 136, in getOSDesc ('ext_list', ext_type len(ext_list)), TypeError: type() argument 2 must be tuple, not _ctypes.PyCStructType

It is complaining about the tuple below. Specifically, line 136 is "('ext_list', ext_type * len(ext_list)),". Very strange.

klass = type(
    'OSDesc',
    OSDescHeader,
    {
        '_fields_': [
            ('ext_list', ext_type * len(ext_list)),
        ],
    },
)
vpelletier commented 6 years ago

It appears we need another typo fix. See below. Right?

Correct. Fixed in d1c09928986fd2df2d9d80deb10f11c6befa21f3 .

It is complaining about the tuple below. Specifically, line 136 is "('ext_list', ext_type * len(ext_list)),". Very strange.

I suspect this is just because it's the last line with an argument to the call which fails. The mistake is actually on the 2nd argument, as in the TypeError message. This should be fixed in 9e8b982db8f2fcdebe119100f72297bf7639b318 .

Sorry for all the trouble you get because of my untested code.

monkey-jsun commented 6 years ago

Not at all. Thank you for your code and patient support.

With the above two fixes, I'm able to move forward a little bit further, but too far. See the error below still in getOSDesc().

Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 220, in main(*sys.argv[1:]) File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 192, in main with FunctionFSTestDevice(path) as function: File "/home/pi/usb-gadget/python-functionfs/functionfs/tests/device.py", line 105, in init os_list.append(functionfs.getOSDesc(0, ext_list)) File "/home/pi/usb-gadget/python-functionfs/functionfs/init.py", line 144, in getOSDesc **kw TypeError: expected BCount instance, got dict

============ I tried to remove one layer of curly bracket as below, but got some other errors.

     wIndex = 4
     kw = {

New error is :
TypeError: expected OSExtCompatDesc_Array_1 instance, got list


Do you have any ideas here? Thanks as always.

vpelletier commented 6 years ago

TypeError: expected BCount instance, got dict

This should be fixed by 28bb78bb3c8ac52178f453cd3bc21aabbd80be81 .

Thanks again for your patience !

monkey-jsun commented 6 years ago

Hi, Vincent,

Thanks for the fix. However, I'm still having the same problem.

After some probing, I'm fairly certain, the problem is due to "ext_list=ext_list,", where left ext_list is of OSExtCompatDesc_Array_1 and right side is an array/list. However, I'm not sure how to solve this due to my limited python experience and your deep programming, :) Do you have any ideas here?

Here is the related code from my application and getOSDesc() code from init.py for your quick review (BTW, according at least one of the web site, the Reserved1 field seems to be 0x1 instead of 0x0. I'm not there to verify it yet.)

def getOSDesc(interface, ext_list):
"""
Return an OS description header.
interface (int)
    Related interface number.
ext_list (list of OSExtCompatDesc or OSExtPropDesc)
    List of instances of extended descriptors.
"""
try:
    ext_type, = {type(x) for x in ext_list}
except ValueError:
    raise TypeError('Extensions of a single type are required.')
if issubclass(ext_type, OSExtCompatDesc):
    wIndex = 4
    kw = {
        'b': OSDescHeaderBCount(
            bCount=len(ext_list),
            Reserved=0,
        ),
    }
elif issubclass(ext_type, OSExtPropDescHead):
    wIndex = 5
    kw = {
        'wCount': len(ext_list),
    }
else:
    raise TypeError('Extensions of unexpected type')
klass = type(
    'OSDesc',
    (OSDescHeader, ),
    {
        '_fields_': [
            ('ext_list', ext_type * len(ext_list)),
        ],
    },
)
return klass(
    interface=interface,
    dwLength=ctypes.sizeof(klass),
    bcdVersion=1,
    wIndex=wIndex,
    ext_list=ext_list,
    **kw
)

=====================
    ext_list = []
    compatDesc = functionfs.OSExtCompatDesc()
    compatDesc.bFirstInterfaceNumber = 0
    compatDesc.Reserved1 = 0x1
    # CompatibleID : U8[8]
    compatDesc.CompatibleID=(ctypes.c_ubyte * 8).from_buffer_copy(b"WINUSB\0\0")
    compatDesc.SubCompatibleID=(ctypes.c_ubyte * 8).from_buffer_copy(b"\0\0\0\0\0\0\0\0")
    compatDesc.Reserved2=(ctypes.c_ubyte * 6).from_buffer_copy(b"\0\0\0\0\0\0")
    ext_list.append(compatDesc)
    osDesc = functionfs.getOSDesc(0, ext_list)  # on interface 0
    # note ending \0 is required by MS spec
    # prop data type 1 : NUL-terminated Unicode String (REG_SZ)
    # see more at https://github.com/pbatard/libwdi/wiki/WCID-Devices
    #propDesc = functionfs.getOSExtPropDesc(1, "DeviceInterfaceGUID\0".encode("utf-16-le"), "{8FE6D4D7-49DD-41E7-9486-49AFC6BFE475}\0".encode("utf-16-le"))
    #ext_list.append(propDesc)
    os_list = []
    os_list.append(osDesc)
vpelletier commented 6 years ago

And one more fix ! a81d3b2ca3ebf73d18ebff147feb49af0e91caea

Thanks for posting your sample code, with above commit it can reach the end successfully. And hopefully, the resulting descriptor may even be correct (I'm ready to believe about anything can go wrong now... Like, how many bugs could possibly live in these 35 lines ?).

(BTW, according at least one of the web site, the Reserved1 field seems to be 0x1 instead of 0x0. I'm not there to verify it yet.)

And that site is correct, the docstring is (was: 6081a5ce3ddea7daccb975114a72a99e52819faf ) wrong. It was wrong where I copied it from (kernel header).

And because it was wrong in the kernel, I even went overzealous (in a very bad way) and caused a regression there by changing the code so it followed the doc. I have few commits in the kernel, and I managed to break a piece of it. Worse, an ABI breakage... shame. That got reverted and I finally fixed the in-header doc (and forgot to update this copy), but I'm still feeling bad about it. I believe in the end kernel forces the value, so that field in userland may not even matter.

monkey-jsun commented 6 years ago

Praise the lord. Hallelujah! It actually worked! At least winusb driver is automatically loaded. I will do more testing to see if everything else works as expected.

Thank you!

One last glitch I hit can be found at http://irq5.io/2016/12/22/raspberry-pi-zero-as-multiple-usb-gadgets/. See quote below.

============ When a new device is attached to a computer for the first time, an operating system […] will request the string descriptor that is at index 0xEE. […] After the operating system requests a Microsoft OS String Descriptor from a device, it creates the following registry key..

The operating system creates a registry entry, named osvc, under this registry key that indicates whether the device supports Microsoft OS Descriptors. If the device does not provide a valid response the first time that the operating system queries it for a Microsoft OS String Descriptor, the operating system will make no further requests for that descriptor.

=======

The solution is to delete the related registry folder as stated below.

When Windows checks for this String Descriptor, one of the first thing that happens is the creation of a registry entry, under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags, that is the concatenation of VID+PID+BCD_RELEASE_NUMBER (This key is never deleted).

vpelletier commented 6 years ago

Finally ! Thanks again for your patience on this embarrassingly (for me) long debugging.

paulosell commented 3 years ago

Hi, Vincent,

Yes, I can confirm that bMS_VendorCode and qw_sign are set in configfs as you suggested.

And functionfs only need to set the following 2 properties:

  • extended feature descriptor, "compatibleID" = "WINUSB"
  • extended property descriptor, where "bPropertyName"="DeviceInterfaceGUID", and "bPropertyData"="{8FE6D4D7-49DD-41E7-9486-49AFC6BFE475}"

@monkey-jsun and @vpelletier any ideia on how doing the above via configfs? By the way, wich USB class were @monkey-jsun trying to configure? I'm trying to configure a cdc acm usb device to act as a WCID to use on Windows 10

vpelletier commented 3 years ago

For the strictly configfs part, your question arrives just in time: these last few days I have been working on automating this part. And I have just pushed the support for writing the 2 OS-descriptor-related configfs files (0208eb9cae2b57dc441cb784e656ffe1a2893b2c). The functionfs part should have been working for a while now (thanks to @monkey-jsun 's patience).

But I still do not know much more than almost 3 years ago, when this bug report was originally opened, on what to actually write in these files, and what function-level descriptors need to be provided.

paulosell commented 3 years ago

@vpelletier thanks for the reply. So, I can assume you dont know how to tell me how to manually set up the configfs/functionfs to make a cdc acm gadget to automatically load WinUSB driver on windows 10, right? I think the modifications on 0208eb9 are not enough for windows to recognize the device as cdc acm and load the winusb driver automatically. The link @monkey-jsun send previously has a RNDIS example on how to set compatible/subcompatible IDs on the device so windows can load its drivers but i cant find an example on cdc acm . I think its a dead end to me :-(

vpelletier commented 3 years ago

So, I can assume you dont know how to tell me how to manually set up the configfs/functionfs to make a cdc acm gadget to automatically load WinUSB driver on windows 10, right?

After my previous reply, I looked around a bit and found a wiki page on WCID devices which seems to have a lot more practical details than most of the other pages I found before put together. I have not read enough to tell if it has the answer you are looking for, but it did look like it had a lot of practical answers.

What I can assist you with, once you know what descriptors you need, is figuring how to get them there with the code of this project (which may involve figuring out kernel-level limitations and cryptic error messages).

I think the modifications on 0208eb9 are not enough for windows to recognize the device as cdc acm and load the winusb driver automatically.

Correct, although they were surely needed to get the values you need where you need them. Assuming I did these right, of course, as I have not tested them (yet ?).

The link @monkey-jsun send previously has a RNDIS example on how to set compatible/subcompatible IDs on the device so windows can load its drivers but i cant find an example on cdc acm .

If you do have an already-working CDC ACM device at hand, maybe you could peek at its descriptors to see what's in there. Maybe by searching these supposed-good values, the spec could be easier to find ?

paulosell commented 3 years ago

Hi there!

I also found the page you mentioned during my research and, altough it offers a very good tutorial on creating the needed descriptor and probably works on a proper microcontroller, I'm not sure on how to create the needed descriptor in a usb linux gadget, like the cdc acm gadget I'm using. Anyway, I'm not gonna waist your time with it. If I find a solution, I'll come back to share with you!

monkey-jsun commented 3 years ago

I eventually got this working, both ACM and our own custom gadget. I always wanted to extract it to create a standalone rpi demo (say, web to rpi chat), but have not got around to do that yet. I attached the patch I committed internally. It has some fixes from @vpelletier and my own, against some old version (around the time of my previous message). Hopefully it is useful for you.

webusb-ffs.patch.txt

vpelletier commented 3 years ago

I'm not sure on how to create the needed descriptor in a usb linux gadget

This is where I should be able to help you.

I attached the patch I committed internally.

Thanks a lot !

Here is a quick review:

py_ffs/*: I confirm all these changes were already applied in python-functionfs. So this part should be working out-of-the-box.

comm_usb.py:

Side note: as you said, this patch is based on rather old code now. In the last few weeks I have made some changes to when endpoints are opened (much heavier use of the python "context manager" pattern), full AIO support (to avoid blocking IO operations and epoll incompatibility without requiring threads).

So depending on what exactly your code relies on it may require some tweaks if you with to update python-libusb. Hopefully not too many, and if you do this then please do not hesitate to report compatibility breaks - some I will probably not be able to do anything for, but I prefer to know how this is used to avoid future compatibility breaks.

paulosell commented 3 years ago

Well, my current scenario is: I have this shell script that is supposed to create my CDC ACM device and configure the microsoft descriptors so windows can load Winusb.sys automatically (wich is my main need here, meaning my device should not ask to install any drivers).

#!/bin/sh
mkdir -p "/sys/kernel/config/usb_gadget/DAF"

echo "0x0525" > "/sys/kernel/config/usb_gadget/DAF/idVendor"
echo "0xa4a7" > "/sys/kernel/config/usb_gadget/DAF/idProduct"
echo "0xEF" > "/sys/kernel/config/usb_gadget/DAF/bDeviceClass"
echo "0x02" > "/sys/kernel/config/usb_gadget/DAF/bDeviceSubClass"
echo "0x01" > "/sys/kernel/config/usb_gadget/DAF/bDeviceProtocol"

# STRINGS
mkdir -p "/sys/kernel/config/usb_gadget/DAF/strings/0x409"
echo "0123456789" > "/sys/kernel/config/usb_gadget/DAF/strings/0x409/serialnumber"
echo "DAF-SC" > "/sys/kernel/config/usb_gadget/DAF/strings/0x409/product"
echo "IFSC" > "/sys/kernel/config/usb_gadget/DAF/strings/0x409/manufacturer"

# CONFIGS
mkdir -p "/sys/kernel/config/usb_gadget/DAF/configs/c.1"
echo "0x80" > "/sys/kernel/config/usb_gadget/DAF/configs/c.1/bmAttributes"
echo "2" > "/sys/kernel/config/usb_gadget/DAF/configs/c.1/MaxPower"
mkdir -p "/sys/kernel/config/usb_gadget/DAF/configs/c.1/strings/0x409"
echo "CDC ACM" > "/sys/kernel/config/usb_gadget/DAF/configs/c.1/strings/0x409/configuration"

# os_desc
echo "1" > "sys/kernel/config/usb_gadget/DAF/os_desc/use"
echo "0x02" > "sys/kernel/config/usb_gadget/DAF/os_desc/b_vendor_code"
echo "MSFT100" > "sys/kernel/config/usb_gadget/DAF/os_desc/qw_sign"
ln -s "/sys/kernel/config/usb_gadget/DAF/configs/c.1" "/sys/kernel/config/usb_gadget/DAF/os_desc/"

# FUNCTIONS
mkdir -p "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0"
echo "WINUSB" > "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0/interface.1/compatible_id"
#echo "\0\0\0\0\0\0\0\0" >  "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0/interface.1/sub_compatible_id"

echo "1" > "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0/interface.1/DeviceInterfaceGUID"
echo "{8FE6D4D7-49DD-41E7-9486-49AFC6BFE475}" >  "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0/interface.1/DeviceInterfaceGUID"

ln -s "/sys/kernel/config/usb_gadget/DAF/functions/acm.GS0" "/sys/kernel/config/usb_gadget/DAF/configs/c.1"

# USB Device Controller (UDC)
basename /sys/class/udc/* > /sys/kernel/config/usb_gadget/DAF/UDC
EOL

Current questions:

Sorry if it looks like dumb questions. I've already tryed different configurations in this file and none seems to work. At first I though my goal was not achievable, but looks like @monkey-jsun managed to achieve it. Does python-functionfs do anything different from what my shell script is capable of doing?

vpelletier commented 3 years ago

From what you posted, here is an example. Depending on exactly how you need to run your functionfs-based functions, some parts will change. For example, if you isolate functions in regular users and subprocesses (which is better for security), GadgetSubprocessManager would be better suited than plain Gadget, and you can use ConfigFunctionSubprocess instances in function_list.

from functionfs.gadget import Gadget
# and in some function:
Gadget(
  idVendor=0x0525,
  idProduct=0xa4a7,
  bDeviceClass=0xef,
  bDeviceSubClass=0x02,
  bDeviceProtocol=0x01,
  lang_dict={
    0x409: {
      "serialnumber": "0123456789",
      "product": "DAF-SC",
      "manufacturer": "IFSC",
    },
    config_list=[{
      "bmAttributes": 0x80,
      "MaxPower": 2,
      "lang_dict": {
        0x409: {
          "configuration": "CDC ACM",
        },
      },
      "function_list": [...], # ConfigFunctionBase instances
    }],
    os_desc={
      "b_vendor_code": 0x02,
      "qw_sign": "MSFT100",
    },
  },
)

The other descriptions belong to the various parameters of functions.Function class (and, likely, subclass).

vpelletier commented 3 years ago

And I forgot to mention: this project (at least in its current state, not sure where I want to go next and how much I can cover before the design no longer makes sense) focusses on functionfs-based functions, so where you would be implementing the whole logic in userland, for when more flexibility is needed than can be achieved with the in-kernel gadgets. As you seem to be using one (f_acm), it may not help you as much as you would expect. Also, as I just translated what you already had, it should not fix anything (it will merely do the same thing your shell code already does, just with python).

vpelletier commented 3 years ago

not sure where I want to go next and how much I can cover before the design no longer makes sense

And I found a way to integrate kernel-implemented functions which I think makes sense in the scope of this project. I reorganised functionfs.gadget code for this: 8e0afba78e20b09358cd7f8036acf09cb4052e43 .