Request: Custom keyboard mapping

[BrockBrinkworth]

Really want custom mapping keyboard controls. I have seen another post about this and someone said it would be too much work but the thread is closed. Keyboard mapping would be amazing :)

[rom1v]

Ref: #149 #505

someone said it would be too much work but the thread is closed

Could you reference the thread?

[BrockBrinkworth]

149 I THOUGHT the bit that said closed meant it was closed... its another thread u linked under it.

How hard is it to make custom buttons like emulators? Because scrcpy would be better than any emulator if it had this mapping. Emulators run like trash compared.

[BrockBrinkworth]

Is anyone still working on scrcpy? If so is this on the road map? Or is there another program or a way to do this myself?

[rom1v]

Is anyone still working on scrcpy?

Yes, on my free time.

If so is this on the road map?

TBH, not really, unless someone wants to work on it (define how to configure key mapping and implement it…).

[BrockBrinkworth]

How the user would configure it: opens an overlay(i.e. steam overlay) and clicks where they want a button and assigns a letter to that button, making that letter click at that spot.

[BrockBrinkworth]

These 2 images show an overlay of Nox Player, pretty much what i am talking about.

[Mehardeep]

hi , am working keyboard mapping for pubg , but i stuck at camera movement (mouse movement) like ctrl+alt to hold and release the mouse anyone plz help??

[BrockBrinkworth]

On this image you can see the writing in the black box, this is what your talking about i am pretty sure. Pretty much i think it locks the mouse in that position and hides it so when you move the mouse only the screen moves and not the cursor. Message again with progress if you can.

[Zireael07]

For my use case, that is controlling a racing game, I was able to workaround this using a bit of Python scripting (I used pynput as pyautogui and keyboard both require running as sudo under Linux).

#!/usr/bin/python2.7

from pynput.mouse import Button, Controller
from pynput.keyboard import Key, Listener

global MOUSE 
MOUSE = Controller()

global LISTEN

global CENTER 
global OFFSET 
global LEFT
global RIGHT
global HEIGHT

# for scrcpy with -2M switch
#HEIGHT = 700 CENTER = 900 OFFSET = 500
# HEIGHT = 500 OFFSET 450 for 1024 -b 4M

# -m 768 -b 4M gives good enough quality while still working (-m 640 doesn't register clicks)
HEIGHT = 650 
CENTER = 950 
OFFSET = 300

LEFT  = CENTER - OFFSET
RIGHT = CENTER + OFFSET

def cust_click(x,y):
    MOUSE.position = (x,y)
    # MOUSE.click(Button.left)
    MOUSE.press(Button.left)

def space():
    cust_click(CENTER,HEIGHT+120) # 150 for -m 1024

def left():
    cust_click(LEFT, HEIGHT)

def right():
    cust_click(RIGHT, HEIGHT)

def on_press(key):
    #print('{0} pressed'.format(
    #    key))

    if key == Key.left:
        left()
    elif key == Key.right:
        right()
    elif key == Key.space:
        space()
    elif key == Key.tab:
        LISTEN.stop()

def on_release(key):
    #print('{0} release'.format(
    #    key))
    if key == Key.right or key == Key.left or key == Key.space:
        MOUSE.release(Button.left)
    # can't use esc because it is the phone 'back' hotkey, at least in snap version
    # if key == Key.esc:
    #     # Stop listener
    #     return False

# Collect events until released
with Listener(
        on_press=on_press,
        on_release=on_release) as LISTEN:
    LISTEN.join()

[anish6400]

Zireal please help me how can i use this code?

[Zireael07]

@AnishKumar0285: You need pynput installed (pip install pynput) and you need to run the script while having the scrcpy open. You'll need to adjust the keys and positions for the game you want to control, obviously, as well as the scrcpy screen's size.

[sukisux]

@AnishKumar0285: You need pynput installed (pip install pynput) and you need to run the script while having the scrcpy open. You'll need to adjust the keys and positions for the game you want to control, obviously, as well as the scrcpy screen's size.

Hello but i have a question i use your script but how can i add new keys and the key when im playing a shot game thet hide the cursor for use only the movement of the mouse, Thanks

[sukisux]

Anyone knows how to add a keys like the a or s key on pynput?

[BrockBrinkworth]

I would also like to know how to properly change and add to this code. I am trying to make it for a shooter game (pubg mobile).

[Zireael07]

Pynput has very good documentation, it should tell you everything you two want to know: https://pythonhosted.org/pynput/index.html

The mouse positions, as I said, have to be eyeballed.

[srevinsaju]

Fixed it in guiscrcpy, https://github.com/srevinsaju/guiscrcpy/blob/master/guiscrcpy/mapper.py It is up on v2.0 pip3 install guiscrcpy It uses pynput, but not as UI intuitive as nox

[BrockBrinkworth]

Thanks, i will try this.

[karousn]

I think, it will be better to have a file external yaml/json from the project (the project just give us an example for the format, like what guiscrcpy do for example) and externalize the work of input_manager.c to do the matching between the event and the correspond on SDL2 action.

[srevinsaju]

Yes @karousn, the only drawback for the guiscrcpy mapping is that it conflicts with the input of characters into a text box, using a mapper key such as Ctrl or Meta may enhance scrcpy 's mode of action. Guiscrcpy manually gives adb shell input and toggles the key position, which is slower than scrcpy's mastermind input method using a tunnel. Will try to implement that into guiscrcpy too :)

[Zireael07]

TC games is windows only, though.

[maximousblk]

If you want an easy implementation, I have a suggestion. I will be really hard for the user to configure, but you can build upon it.

Simplest thing to do is use a config file, preferably json, in which you can map clicks and movements simulation. Why would this be hard? because you have to map the keys and movements using pixels for measurement.

Now if you want a hard implementation but easy for users to configure, here's what you need to do.

1. create a GUI for the whole thing
2. In the gui have a seperate tab for mapping
3. In that tab user would take a screenshot (or live) of the situation which they want it for.
4. Then map the keys according to the screenshot
5. Launch scrcpy and press a special keystroke which would activate the mapping.

Just a suggestion.

this extends #1453

[vincentyliu2001]

I would like to add that an easy way to implement key mappings would be to use an AutoHotKey script. Make sure you run it as administrator so key inputs are not fed into scrcpy itself but actually fed into the script.

[hugonh]

Any way to key map?

[hugonh]

I would like to add that an easy way to implement key mappings would be to use an AutoHotKey script. Make sure you run it as administrator so key inputs are not fed into scrcpy itself but actually fed into the script.

can you help me with autohotkey as key mapper?

[vincentyliu2001]

I would like to add that an easy way to implement key mappings would be to use an AutoHotKey script. Make sure you run it as administrator so key inputs are not fed into scrcpy itself but actually fed into the script.

can you help me with autohotkey as key mapper?

Autohotkey scripting language is relatively easy to pick up, especially for a simple script. Here is their guide that contains all the information you would ever need to make a script and explains things better than I ever could. https://www.autohotkey.com/docs/Tutorial.htm

[rom1v]

Here is a simple proposal to fix both #712 and #1302 with a single --map option (not implemented yet):

scrcpy --map 'z=click:200:300,a=click:500:400,space=click:780:40,rclick=inject,mclick=back'

• pressing z will inject a left-click at x=200, y=300
• pressing a will inject a left-click at x=500, y=400
• pressing SPACE will inject a left-click at x=780, y=40
• right-click will be injected to the device (instead of pressing BACK)
• middle-click will inject BACK (default is HOME)

The coordinates are expressed in device screen resolution (so that they are independant of the --max-size option) in natural orientation (typically portrait for a phone, landscape for a tablet, like the --crop option).

What do you think? Do you see any problem with it?

[hugonh]

Qtscrcpy does that well.

[rom1v]

OK :+1: will read that: https://github.com/barry-ran/QtScrcpy/blob/master/docs/KeyMapDes.md

The coordinates are expressed in device screen resolution

In fact, they should be totally relative instead (between 0 and 1), so that they can be reused across devices.

[rom1v]

Now I think that #1302 could be fixed quickly separately (--right-click=inject --middle-click=back for example).

[renanwp2]

There is the great project called Panda Mouse Pro. It is a non-free and paid apk, unfortunately, but I would suggest, kindly and desperately, someone implementing the mouse and keyboard input to work with it. I can't imagine if it's possible to achieve a better results than using this apk. I have heard that the SupremeGamers have achieved similar results on Android x86.

P.S.: For now, the unique way we can play games without dual booting is using an USB Hub on smartphone, run some app like Panda Mouse Pro, Octopus or Tencent Game Stick and to mirror the screen to the computer/laptop and to play with it. It will work reasonably well. =) As far I know, there is no way to play games on Linux other than that. I have tried Genymotion, Virtual Machine (Android x86, Phoenix OS, Bliss OS), Anbox, Scrcpy (without a HUB) and scripts like Zireael07 have done and none of then is satisfactory. I think, for now, the unique possibility for playing without dual booting is doing that.

Super P.S.: Ok! I give Qtscrcpy a shot and after one thousand's tries, I was able to compile QtScrcpy. It works as intended and looks good as replacement for Scrcpy in games, while the things does not get better. It was needed to revert the patch (cbed591aed000c1e969445480cbb353a52e1139d) that updated the Qt to the version 5.15.1 on Debian Buster and Ubuntu 20.04. Since I'm not so fluent on Github engine, I dont know If I reverted everything right, but my compilation neither have shown error nor warnings during compilation after that. =) Maybe I will upload my build on Github. I compiled on Debian, and I guess it will work on all Debian based distros, but I'm not sure, since the converse is not usually true.

Long long time after last P.S.: Some comments on using USB Hub (OTG Cable) plugged on smartphone with Scrcpy. It's not a possible thing to accomplish, since for using USB Hub (OTG Cable) your phone needs be the host. About playing with Qtscrcpy, it's not a good option for all games, since some games have problems with his implementation of KMT_STEER_WHEEL (ASWD keys), cos some of them requires some delay after first click on the screen to take into account the second click on the screen. Basically, there is no perfect way for playing games up to now. At least, PUBG is working as far I could test, but it's the only one I know that works with Qtscrcpy..

[renanwp2]

Here is a simple proposal to fix both #712 and #1302 with a single --map option (not implemented yet):

scrcpy --map 'z=click:200:300,a=click:500:400,space=click:780:40,rclick=inject,mclick=back'

* pressing z will inject a left-click at x=200, y=300

* pressing a will inject a left-click at x=500, y=400

* pressing SPACE will inject a left-click at x=780, y=40

* right-click will be injected to the device (instead of pressing BACK)

* middle-click will inject BACK (default is HOME)

The coordinates are expressed in device screen resolution (so that they are independant of the --max-size option) in natural orientation (typically portrait for a phone, landscape for a tablet, like the --crop option).

What do you think? Do you see any problem with it?

Yes. On some games, it's needed some delay after the first click to the second click work. I think a implementation of multi click and delay would be good for WASD keys. Of course, the last click needs to remain happening after the other ones for WASD keys to work. You can test this behavior trying to simulate mouse click with xbindkeys for playing CoD or Free Fire.

[FinlayDaG33k]

What do you think? Do you see any problem with it?

It would be nice to maybe have it load a file (eg. a JSON/Yaml/XML- pick your poison) so you can have different maps without having to redo big command line arguments. Additionally, it'd need slides as well (most games requires the player to slide the "joystick" to move their characters).

[guanzhangrtk]

I did a quick proof-of-concept using Python and ncurses and was able to come up with a simple terminal-based keymapper which requires adb running. Key mapping configurations are stored in per app/per resolution json files and we may be able to re-use the same mapping file if the resolution scales. I wonder if people will find it useful or would it be considered "re-inventing the wheel". It currently supports:

• taps
• swipes
• macros (sequential combination of the above inputs)

The following are TODOs:

• Playback recorded macros

Latency won't be great because it relies on adb but I don't see why it can't be integrated with scrcpy like this proposal or just use something more low-level to interact with Android.

Thoughts, comments? Thanks!

EDIT: The beauty of it being a terminal app is that I can control the game running on my Android device remotely even from an iPad running iSH :)

[onguarde]

found autohotkey works surprisingly well, you should definitely try that. all my capslock + jkli keys mapping works perfectly.

i definitely prefer scrcpy vs directly connecting my keyboard to the android phone.

; Homerow
; Home/End Keys
Capslock & h::send {Home}
Capslock & `;::send {End}
; escape code for ; is for commenting - -> ` -> One tilda ~

;
; Arrow keys
; Use blind / make it work with combo keys (eg. press shift to ctrl+shift+arrow keys)

; uo
; left
Capslock & u::Send {Blind}{Left DownTemp}
Capslock & u up::Send {Blind}{Left Up}

; right
Capslock & o::Send {Blind}{Right DownTemp}
Capslock & o up::Send {Blind}{Right Up}

; ijkl
; ctrl + left
Capslock & j::Send {Blind}{Ctrl down}{Left DownTemp}
Capslock & j up::Send {Blind}{Left Up}{Ctrl up}

; down
Capslock & k::Send {Blind}{Down DownTemp}
Capslock & k up::Send {Blind}{Down Up}

; up
Capslock & i::Send {Blind}{Up DownTemp}
Capslock & i up::Send {Blind}{Up Up}

; ctrl + right
Capslock & l::Send {Blind}{Ctrl down}{Right DownTemp}
Capslock & l up::Send {Blind}{Right Up}{Ctrl up}

[inxomnyaa]

That repo gives a 404 for me - i wanted to suggest that you check for the name of the app, see https://stackoverflow.com/questions/15992342/how-to-detect-whether-a-process-or-application-is-running-and-terminate-the-prog/16000270#16000270

[renanwp2]

I have uploaded the QtScrcpy binaries on my page. You just need to clone my repository to have it working .This project is based upon scrcpy with "all" wanted features. It should work on all Debian-based systems (Compiled on Debian Buster). If it does not run, try running on terinal ldd QtScrcpy to track possible libraries lacking on your system. I will test this version to see if it's possible to play Free Fire, but supposedly it's possible, after this fix.

P.S.: It's working with Free Fire. I think the fix is fine. Your turn to test it I won't test it anymore so soon.

[nutlost]

I have uploaded the QtScrcpy binaries on my page. You just need to clone my repository to have it working .This project is based upon scrcpy with "all" wanted features. It should work on all Debian-based systems (Compiled on Debian Buster). If it does not run, try running on terinal ldd QtScrcpy to track possible libraries lacking on your system. I will test this version to see if it's possible to play Free Fire, but supposedly it's possible, after this fix.

P.S.: It's working with Free Fire. I think the fix is fine. Your turn to test it I won't test it anymore so soon.

"qtscrcpy" I can create all buttons myself except this one. I can't create move mouse button. I can't create more than one "Smalleyes" button. What should I do?

[MarceloMachadoxD]

guys sorry for asking here but i dont want create a new thread but... did you be able to run mapping keyboard to screenpress positions? i want to configure it to play genshin impact but i really dont understand well the suggestions with phyton theres an resumed area on how to configure it?

[guanzhangrtk]

Perhaps you guys can look at this project. It currently works with Bliss OS (Android-x86) however I don't see how it can't be adapted to real mobile devices...??

[renanwp2]

guys sorry for asking here but i dont want create a new thread but... did you be able to run mapping keyboard to screenpress positions? i want to configure it to play genshin impact but i really dont understand well the suggestions with phyton theres an resumed area on how to configure it?

Please, see the project QtScrcpy. Is it easy to install? Absolutely not. Good luck. On Linux, there is no easy way to play Android games.

[MarceloMachadoxD]

guys sorry for asking here but i dont want create a new thread but... did you be able to run mapping keyboard to screenpress positions? i want to configure it to play genshin impact but i really dont understand well the suggestions with phyton theres an resumed area on how to configure it?

Please, see the project QtScrcpy. Is it easy to install? Absolutely not. Good luck. On Linux, there is no easy way to play Android games.

Thanks so much friend, you're a friend

[king-om]

TBH, not really, unless someone wants to work on it (define how to configure key mapping and implement it…).

may be, we can make a script for the android device to listen to the input through the tunnel or something like that...! I don't know much but I am a student and learning about development...

[hugonh]

do you guys playing watching by mobile screen or pc's monitor? for me, i cant handle input lag, i tried to manipulate screen resolution with qts and bitrate, but fps drops a lot

[rom1v]

with qts

What is qts?

fps drops a lot

That's not expected.

[luisalvarado]

Sorry for being late at the party here which I find very interesting for mimicking touch movements, dragging and such.

I am using Ubuntu 22.10 and 23.04 with scrcpy 2.0. Which method would be recommended to easily map for example the WASD keys to a touchscreen area for the up, down, left, right movements.

[luisalvarado]

If it helps for research purposes, the xdotool is the one I am using to more or less mimic this:

Currently it is able to drag the center of the virtual joystick (What I call the center position) and drag it to the corresponding X/Y coordinates for the UP | DOWN | LEFT | RIGHT positions. Depending if I hold the SHIFT, it will go longer to any direction to mimic the running speed, or move less to the sides to mimic the stealth/slower movement.

The only problems are the following:

1. If I change the sizes of the window, then I gotta redo the X Y positions for all combinations. I am looking for a way to make the X Y positions proportional to the window size. If genymobile does indeed include this, the window size would need to be taken into consideration when resizing or moving the window around. In the meantime I am thinking this out.

2. An option to drag the touch effect around. With this I mean, If I press the UP + LEFT key , it will click and hold on the center, then drag it to the Upper / Left side. But what happens if I want to change from Up to Right. I would want the effect to start where I left the touch event and then move to the Upper / Right corner of the center position, but without having to release first, then touch the center, then drag to the upper right side. This is because, in-game, the character would pause for a second to accomplish this, instead of naturally moving from left to right.

3. Use the mouse to "look around" without needing to use the mouse to click. So that, the keyboard would move the character around the map, while the mouse would turn the character in any angle.

4. Map the mouse buttons to the action buttons along with mapping keyboard keys to it as well (at the same time), so at any moment I can click the mouse, or press a key and it would execute the action button in game.

This is what I am working on with the xdotool and thinking my way through. The github mentioned above did not help at all btw.

[niksingh710]

If it helps for research purposes, the xdotool is the one I am using to more or less mimic this:

Currently it is able to drag the center of the virtual joystick (What I call the center position) and drag it to the corresponding X/Y coordinates for the UP | DOWN | LEFT | RIGHT positions. Depending if I hold the SHIFT, it will go longer to any direction to mimic the running speed, or move less to the sides to mimic the stealth/slower movement.

The only problems are the following:

1. If I change the sizes of the window, then I gotta redo the X  Y positions for all combinations. I am looking for a way to make the X Y positions proportional to the window size. If genymobile does indeed include this, the window size would need to be taken into consideration when resizing or moving the window around. In the meantime I am thinking this out.

2. An option to drag the touch effect around. With this I mean, If I press the UP + LEFT key , it will click and hold on the center, then drag it to the Upper / Left side. But what happens if I want to change from Up to Right. I would want the effect to start where I left the touch event and then move to the Upper / Right corner of the center position, but without having to release first, then touch the center, then drag to the upper right side. This is because, in-game, the character would pause for a second to accomplish this, instead of naturally moving from left to right.

3. Use the mouse to "look around" without needing to use the mouse to click. So that, the keyboard would move the character around the map, while the mouse would turn the character in any angle.

4. Map the mouse buttons to the action buttons along with mapping keyboard keys to it as well (at the same time), so at any moment I can click the mouse, or press a key and it would execute the action button in game.

This is what I am working on with the xdotool and thinking my way through. The github mentioned above did not help at all btw.

This seems promising can you do one with wtype and put it on git?

[luisalvarado]

Thanks, I will research what wtype is and if it can solve an issue with multiple keys being pressed.

[B4tiste]

How can I make scrcpy use ESCAPE for BACK instead of using the Right Click ?