search

Genymobile / scrcpy

Display and control your Android device
Apache License 2.0
111.39k stars 10.67k forks source link

HID support on Windows #2773

Closed rom1v closed 2 years ago

rom1v commented 2 years ago

2632 added support for HID keyboards, using libusb. It is currently only supported on Linux (because it just works).

Here is a branch where libusb support is enabled for Windows (and the libusb dependency added to the release/build process): libusb-windows

The problem we face is that the access to the device is always denied:

> scrcpy --otg -Vdebug
…
DEBUG: Open USB device xxxx:xxxx: libusb error: Access denied (insufficient permissions)
ERROR: Could not find any USB device

If you know how to make it work, your help is welcome.

Checkout the libusb-windows branch, then follow the BUILD instructions

For simplicity, here is a prebuilt server for this branch (so that you just have to build the client, it's explained in the BUILD instructions):

chldbwnstm commented 2 years ago

Hi rom,

If I use the default mode and click x=200,y=300 with my mouse on scrcpy, do you use adb shell input tap command to send the event?

rom1v commented 2 years ago

do you use adb shell input tap command to send the event?

Scrcpy does not call adb shell input tab, but they both use the same mechanism to inject events. https://github.com/Genymobile/scrcpy/blob/master/DEVELOP.md#input-events-injection

vsutardja commented 2 years ago

Using this project as a base, I've been able to put together a quick and dirty proof of concept that can inject HID mouse motion events to my Pixel 6 from Windows 10.

Console output looks like this when run:

C:\Users\vsutardja\Downloads>linux-adk -d 18d1:4ee7
Looking for device 18d1:4ee7
Device supports AOA 2.0!
Sending identification to the device
Turning the device in Accessory mode
Found accessory 18d1:2d05
Sending identification to the device
Closing USB device

On the phone, you'll see a dialog about an accessory app and then after the last Sending identification to the device message, a cursor should appear and move down and to the left 16 times before stopping.

There are still some issues to iron out though, namely that I can't get it to coexist with an adb connection. I have to run adb kill-server if the daemon's already running in order for libusb to be able to open the device, while adb devices finds nothing when the program is already running.

The AOAv1 documentation mentions that PIDs with ADB support expose an interface that handles ADB communication. For AOAv2 these are PIDs 0x2D03 and 0x2D05. By comparing the descriptors from before and after entering accessory mode (listed below), I can see that my Pixel 6 with accessory mode PID 0x2D05 has what looks like an ADB interface as its last (bInterfaceNumber 3), but how to make use of it is something I have yet to explore.

Hope this can be of some use!

Before, with just one interface, which I assume is for ADB:

  Configuration:
    wTotalLength:        32
    bNumInterfaces:      1
    bConfigurationValue: 1
    iConfiguration:      0
    bmAttributes:        80h
    MaxPower:            250
    **Interface:
      bInterfaceNumber:   0
      bAlternateSetting:  0
      bNumEndpoints:      2
      bInterfaceClass:    255
      bInterfaceSubClass: 66
      bInterfaceProtocol: 1
      iInterface:         4
      Endpoint:
        bEndpointAddress: 01h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0
      Endpoint:
        bEndpointAddress: 81h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0**

After, with 4 interfaces, the last of which has a very similar descriptor to the ADB descriptor above:

  Configuration:
    wTotalLength:        156
    bNumInterfaces:      4
    bConfigurationValue: 1
    iConfiguration:      0
    bmAttributes:        80h
    MaxPower:            250
    Interface:
      bInterfaceNumber:   0
      bAlternateSetting:  0
      bNumEndpoints:      2
      bInterfaceClass:    255
      bInterfaceSubClass: 255
      bInterfaceProtocol: 0
      iInterface:         4
      Endpoint:
        bEndpointAddress: 81h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0
      Endpoint:
        bEndpointAddress: 01h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0
    Interface:
      bInterfaceNumber:   1
      bAlternateSetting:  0
      bNumEndpoints:      0
      bInterfaceClass:    1
      bInterfaceSubClass: 1
      bInterfaceProtocol: 0
      iInterface:         0
    Interface:
      bInterfaceNumber:   2
      bAlternateSetting:  0
      bNumEndpoints:      0
      bInterfaceClass:    1
      bInterfaceSubClass: 2
      bInterfaceProtocol: 0
      iInterface:         0
    Interface:
      bInterfaceNumber:   2
      bAlternateSetting:  1
      bNumEndpoints:      1
      bInterfaceClass:    1
      bInterfaceSubClass: 2
      bInterfaceProtocol: 0
      iInterface:         0
      Endpoint:
        bEndpointAddress: 82h
        bmAttributes:     0dh
        wMaxPacketSize:   256
        bInterval:        4
        bRefresh:         0
        bSynchAddress:    0
    Interface:
      bInterfaceNumber:   3
      bAlternateSetting:  0
      bNumEndpoints:      2
      bInterfaceClass:    255
      bInterfaceSubClass: 66
      bInterfaceProtocol: 1
      iInterface:         4
      Endpoint:
        bEndpointAddress: 02h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0
      Endpoint:
        bEndpointAddress: 83h
        bmAttributes:     02h
        wMaxPacketSize:   512
        bInterval:        0
        bRefresh:         0
        bSynchAddress:    0
rom1v commented 2 years ago

Hi @vsutardja,

Thank you for our investigations.

By looking at the source code of linux-adk, the principle is similar to what scrcpy does.

The problem I encounter is that I always get "Access denied" when opening a device with libusb on Windows. I updated the initial post of this thread to provide more details.

Could you try this branch (libusb-windows), please?

The AOAv1 documentation mentions that PIDs with ADB support expose an interface that handles ADB communication. For AOAv2 these are PIDs 0x2D03 and 0x2D05. By comparing the descriptors from before and after entering accessory mode (listed below), I can see that my Pixel 6 with accessory mode PID 0x2D05 has what looks like an ADB interface as its last (bInterfaceNumber 3), but how to make use of it is something I have yet to explore.

Scrcpy does not rely on vid:pid to select the device:

Instead, it just considers all the available USB devices having a serial and which could be opened (and this mechanism works pretty well when the devices could be opened).

vsutardja commented 2 years ago

Hi Romain,

Thanks for taking a look!

Interestingly enough, the libusb-windows branch just works for me - on two different computers connected to two different devices, no less.

On a Windows 10 PC running build 19041.1.amd64fre.vb_release.191206-1406 connected to a 2019 Amazon Fire 7:

❯ scrcpy.exe --otg -Vdebug
scrcpy 1.22 <https://github.com/Genymobile/scrcpy>
DEBUG: USB device found:
DEBUG:     -->   <SERIALNO> (1949:03c8)  Amazon KFMUWI
INFO: USB device: <SERIALNO> (1949:03c8) Amazon KFMUWI
WARN: libusb does not have hotplug capability
WARN: Could not register USB device disconnection callback
DEBUG: Starting AOA thread
DEBUG: Using icon (portable): C:\Users\Victor\Projects\scrcpy\release-v1.22-54-g26ee63d9\icon.png

On a Macbook Pro running Windows 10 through Boot Camp, also build 19041.1.amd64fre.vb_release.191206-1406, connected to a Pixel 6:

C:\Users\vsutardja\Downloads\scrcpy>scrcpy.exe --otg -Vdebug
scrcpy 1.22 <https://github.com/Genymobile/scrcpy>
DEBUG: Open USB device 05ac:8406: libusb error: Operation not supported or unimplemented on this platform
DEBUG: USB device found:
DEBUG:     -->     <SERIALNO> (18d1:4ee7)  Google Pixel 6
INFO: USB device: <SERIALNO> (18d1:4ee7) Google Pixel 6
WARN: libusb does not have hotplug capability
WARN: Could not register USB device disconnection callback
DEBUG: Starting AOA thread
DEBUG: Using icon (portable): C:\Users\vsutardja\Downloads\scrcpy\icon.png
DEBUG: HID keyboard state synchronized

I have a OnePlus 5T that I can try as well later tonight.

This might be a dumb question but did you make sure to run adb kill-server before starting scrcpy? I get the same error if the daemon is still running:

C:\Users\vsutardja\Downloads\scrcpy>adb get-serialno
* daemon not running; starting now at tcp:5037
* daemon started successfully
<SERIALNO>

C:\Users\vsutardja\Downloads\scrcpy>scrcpy.exe --otg -Vdebug
...
DEBUG: Open USB device 18d1:4ee7: libusb error: Access denied (insufficient permissions)
ERROR: Could not find any USB device

I think there may be a slight misconception here, if I'm understanding correctly:

You can see in the working debug output above that neither the Amazon Fire 7 nor the Pixel 6 have a VID:PID matching the header file definitions either. This is expected - by default a device just presents its manufacturer VID:PID, not an AOA-compliant one. Not only is this manufacturer VID:PID interface sufficient to query AOA support and start accessory mode, it's actually the sole case where we even need to do so. That is, if the VID:PID already matches the AOA spec, then the device is guaranteed to support AOA, as it is already operating in accessory mode.

You're certainly right that USB debugging needs to be enabled on the device (at least that's been my experience so far). However, an actual adb connection opens and occupies the device, preventing other processes such as scrcpy from doing so.

rom1v commented 2 years ago

This might be a dumb question but did you make sure to run adb kill-server before starting scrcpy? I get the same error if the daemon is still running

Wow, indeed! :tada:

That's was my problem for years! I had never been able to connect to an Android device using libusb on Windows.

However, this is a huge restriction: for example scrcpy --hid-keyboard should work even when mirroring is enabled (i.e. not in OTG mode, so adb is necessarily enabled) :confused:

vsutardja commented 2 years ago

However, this is a huge restriction

Though I'm personally happy just using it in OTG mode, I completely agree.

The docs mention that at least for PIDs 2D03 and 2D05, there should be an ADB interface available even in accessory mode, which I think is borne out by the descriptors included in my first comment.

How to actually use said interface is another issue entirely. Since only one process can hold the device handle open at any given time, the only option I can think of would be to re-implement the ADB protocol through libusb within scrcpy, which I could easily understand being considered out-of-scope for this project.

However, if there is any interest in pursuing this approach, a quick search did turn up a bit of prior art - see cstyan's protocol documentation and cgutman's Java library implementation - although I'm not sure how up-to-date they are.

rom1v commented 2 years ago

the only option I can think of would be to re-implement the ADB protocol through libusb within scrcpy

Many developers use scrcpy for Android development, so adb commands could still be executed while scrcpy -K is running.

FYI, I asked on libusb: https://github.com/libusb/libusb/issues/1065.

rom1v commented 2 years ago

I opened a PR to support --otg on Windows: https://github.com/libusb/libusb/wiki/Windows#driver-installation

We could investigate using libusbk to support HID keyboard and mouse even if adb is running (to support HID + mirroring on Windows): https://github.com/libusb/libusb/wiki/Windows#driver-installation

vsutardja commented 2 years ago

I had the same idea re: leveraging libusbK, since it purports to support concurrent connections from multiple applications.

A quick initial test replacing the driver resulted in scrcpy otg mode still working, but adb not being able to find the device at all.

image

C:\Users\vsutardja\Downloads>adb devices
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached

C:\Users\vsutardja\Downloads\scrcpy>scrcpy --otg -Vdebug
scrcpy 1.22 <https://github.com/Genymobile/scrcpy>
DEBUG: Open USB device 05ac:8406: libusb error: Operation not supported or unimplemented on this platform
DEBUG: USB device found:
DEBUG:     -->     <SERIALNO> (18d1:4ee7)  Google Pixel 6
INFO: USB device: <SERIALNO> (18d1:4ee7) Google Pixel 6
WARN: libusb does not have hotplug capability
WARN: Could not register USB device disconnection callback
DEBUG: Starting AOA thread
DEBUG: Using icon (portable): C:\Users\vsutardja\Downloads\scrcpy\icon.png
DEBUG: HID keyboard state synchronized
DEBUG: User requested to quit
DEBUG: quit...
AlynxZhou commented 2 years ago

I think I tried every driver for my Galaxy S9+ and none works, driver except WinUSB are not stable yet.

vsutardja commented 2 years ago

Going back to the idea of adb-within-scrcpy, I think theoretically it should be possible to still use adb commands from the shell or a script, but I'll admit it's kludgy.

If I'm interpreting this document correctly, the adb command line program just puts in requests to an adb server, which is itself a separate entity that handles all the actual I/O to the adb daemons running on the devices. If we could bundle an adb server into scrcpy and get the CLI client(s) to talk to it instead of spinning up their own server, that would solve the problem of requiring two separate applications holding the same USB interface.

Here's a supposed drop-in server replacement implemented in Go: https://github.com/danielpaulus/go-adb

rom1v commented 2 years ago

@vsutardja That's interesting, but then closing scrcpy would close the adb daemon, and running several scrcpy instances would not work correctly.

AndroidDeveloperLB commented 2 years ago

I'm not sure what this thread is about. Is it about adding support for Unicode keys on Windows? If so, the latest version 1.23 doesn't work for me. I tried to type Hebrew characters. It just says it fails with them. By seeing "OTG" being mentioned, maybe this means it works only if I connect the keyboard to the device, and not via Windows ? Will it ever change? Is it something that Android can change to allow it? Or just Windows? Or any of them?

XayUp commented 1 year ago

Almost 1 year ago this topic but, when I read and understood "Use Scrcpy mirroring while using --otg, on Windows", I remembered that this is possible. It's a gimmick but it works. I created two separate ADB folders. One with "adb.exe" and another with "adb1.exe" (The name doesn't matter, it's just to differentiate one from the other). In one of the ADBs I ran OTG mode (Obviously in USB mode) and in another adb I ran Scrcpy mirroring via WIFI. And it worked perfectly.

Of course, there's that little Scrcpy window when in --otg mode that can't stay off the screen. As said, this is a gimmick.

Tryanks commented 1 year ago

Almost 1 year ago this topic but, when I read and understood "Use Scrcpy mirroring while using --otg, on Windows", I remembered that this is possible. It's a gimmick but it works. I created two separate ADB folders. One with "adb.exe" and another with "adb1.exe" (The name doesn't matter, it's just to differentiate one from the other). In one of the ADBs I ran OTG mode (Obviously in USB mode) and in another adb I ran Scrcpy mirroring via WIFI. And it worked perfectly.

Of course, there's that little Scrcpy window when in --otg mode that can't stay off the screen. As said, this is a gimmick.

I like this idea, it does seem to work. Also, it makes me support the idea of "making the server a standalone installable app". If the server is app independent, then the streaming data communication can be reused for USB connection via AoAv1. I understand that this is not an easy task to do in the near future, but I would like to hear from @rom1v , do you think this is a viable way forward?