Press Key issue with combined keys

[Mavnus04]

I am having an issue pressing several keys on my application.

I can send a single key press with t'~' without an issue. But when I try to send ALT+SHIFT+- or ALT+SHIFT+=, it seems to respond only to the keys - or =. Yes, this key combo normally works on the application. As you can see in my example, I tried sending it as t'' or s'' or assign it as a string, then sending it.

Show the info bar on the unity window 
    [Tags]  unity  
    Log to console  ...
    ${gamename} =  Set Variable  mygame
    Attach application by name  ${gamename}
    FlaUILibrary.Focus   /Window[@Name='${gamename}']

    FlaUILibrary.Press Key  t'~'
    Sleep  3s
    FlaUILibrary.Press Key  t'~'
    Sleep  1s

    # 'Half Time Scale': 'Alt+Shift+(minus)',
    FlaUILibrary.Press Key  t'ALT+SHIFT+-'
    Sleep  1s
    FlaUILibrary.Press Key  s'ALT+SHIFT+-'
    Sleep  1s
    ${halfTime} =  Convert To String  t'ALT+SHIFT+-'
    FlaUILibrary.Press Key  ${halfTime}
    Sleep  1s
    ${halfTime2} =  Convert To String  s'ALT+SHIFT+-'
    FlaUILibrary.Press Key  ${halfTime2}
    Sleep  3s

    #'Double Time Scale': 'Alt+Shift+(equals)',
    FlaUILibrary.Press Key  t'ALT+SHIFT+='
    Sleep  1s
    FlaUILibrary.Press Key  s'ALT+SHIFT+='
    Sleep  1s
    ${doubleTime} =  Convert To String  t'ALT+SHIFT+='
    FlaUILibrary.Press Key  ${doubleTime}
    Sleep  1s
    ${doubleTime2} =  Convert To String  s'ALT+SHIFT+='
    FlaUILibrary.Press Key  ${doubleTime2}

Current version:

C:\Windows\system32>pip show robotframework-flaui
Name: robotframework-flaui
Version: 1.7.3

Related... I don't understand the difference between the t'' and s'', can you explain more with an example? Finally, can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

[Mavnus04]

I also have combo commands for ALT+SHIFT+BACK and ALT+SHIFT+0 (The '0' key on the top of the alphanumeric keyboard.), but these don't work either.

[Nepitwin]

Code looks quite good that it should work. I will investigate on Keyboard module where the problem comes.

I think it would be a good advice to extend keyboard tests by a gui element which send the response from entered input to a input field to verify if keyboard bindings work as expected.

[Mavnus04]

@Nepitwin Any update? Or could you at least answer my other questions for now?

Related... I don't understand the difference between the t'' and s'', can you explain more with an example? Finally, can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

[Nepitwin]

@Mavnus04 oh i don't see your question.

t'' syntax is used for a text usage and s'' by syntax

For example if you want to input into a text field you should use t"Your Text Here"

By a shortcut you use s"CTRL+A" this type of can be mixxed up by a list

For example

${EXP_VALUE_OVERRIDE_INPUT_TEXT} = Override @{KEYBOARD_INPUT_OVERRIDE_TEXT} s'CTRL+A' t'${EXP_VALUE_OVERRIDE_INPUT_TEXT}'

[Nepitwin]

What i can also see from docs

    The following keys are supported for usage as a part of key_combination:
    | LBUTTON     |     Left mouse button                           |
    | RBUTTON     |     Right mouse button                          |
    | CANCEL      |     Control-break processing                    |
    | MBUTTON     |     Middle mouse button (three-button mouse)    |
    | XBUTTON1    |     Windows 2000/XP: X1 mouse button            |
    | XBUTTON2    |     Windows 2000/XP: X2 mouse button            |
    | BACK        |     BACKSPACE key                               |
    | TAB         |     TAB key                                     |
    | CLEAR       |     CLEAR key                                   |
    | ENTER       |     ENTER key                                   |
    | SHIFT       |     SHIFT key                                   |
    | CTRL        |     CTRL key                                    |
    | ALT         |     ALT key                                     |
    | CAPITAL     |     CAPITAL key                                 |
    | PAUSE       |     PAUSE key                                   |
    | ESCAPE      |     ESC key                                     |
    | ESC         |     ESC key                                     |
    | SPACE       |     Blank space key                             |
    | NEXT        |     Next key                                    |
    | END         |     END key                                     |
    | HOME        |     HOME key                                    |
    | LEFT        |     LEFT ARROW key                              |
    | RIGHT       |     RIGHT ARROW key                             |
    | UP          |     UP ARROW key                                |
    | DOWN        |     DOWN ARROW key                              |
    | SELECT      |     SELECT key                                  |
    | PRINT       |     PRINT key                                   |
    | EXECUTE     |     EXEC key                                    |
    | INSERT      |     INS key                                     |
    | DELETE      |     DEL key                                     |
    | HELP        |     HELP key                                    |
    | 0 - 9       |                                                 |
    | A - Z       |                                                 |
    | F1 - F12    |                                                 |
    | LWIN        |     Left Windows key                            |
    | RWIN        |     Right Windows key                           |
    | APPS        |                                                 |
    | SLEEP       |                                                 |
    | MULTIPLY    |     '*' key                                     |
    | ADD         |     '+' key                                     |
    | SEPARATOR   |                                                 |
    | SUBTRACT    |                                                 |
    | DECIMAL     |                                                 |
    | DIVIDE      |                                                 |

You want to create a shortcut so s is correct for my opinion.

So the input from your sample code should work i will investigate your example until next week.

[Mavnus04]

@Nepitwin One more question related to the documentation while we wait for your investigation: Can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

[Nepitwin]

This documentation is outdated. The only thing that the Keyword is doing that Press Key can be used by XPATH optional to focus the element before a keyboard execution is triggered.

So it's quite similar like your code

Focus Press Keys

Is equivalent to

Press Keys

I will now extend the test gui by a keyboard testing view. Which displays the keyboard inputs and extend the testcases by multiple keyboard bindings then i should find easily the issue.

[Nepitwin]

Ah now i see what the issue is. You try to use the VirtuaKeyShort with simple mapping. But what i can see is that +,- and equals is not supported from current Key mapping to flaui

Keys = {
    "LBUTTON": VirtualKeyShort.LBUTTON,  # Left mouse button
    "RBUTTON": VirtualKeyShort.RBUTTON,  # Right mouse button
    "CANCEL": VirtualKeyShort.CANCEL,  # Control-break processing
    "MBUTTON": VirtualKeyShort.MBUTTON,  # Middle mouse button (three-button mouse)
    "XBUTTON1": VirtualKeyShort.XBUTTON1,  # Windows 2000/XP: X1 mouse button
    "XBUTTON2": VirtualKeyShort.XBUTTON2,  # Windows 2000/XP: X2 mouse button
    "BACK": VirtualKeyShort.BACK,  # BACKSPACE key
    "TAB": VirtualKeyShort.TAB,  # TAB key
    "CLEAR": VirtualKeyShort.CLEAR,  # CLEAR key
    "ENTER": VirtualKeyShort.ENTER,  # ENTER key
    "SHIFT": VirtualKeyShort.SHIFT,  # SHIFT key
    "CONTROL": VirtualKeyShort.CONTROL,  # CTRL key
    "CTRL": VirtualKeyShort.CONTROL,
    "ALT": VirtualKeyShort.ALT,
    "CAPITAL": VirtualKeyShort.CAPITAL,
    "PAUSE": VirtualKeyShort.PAUSE,
    "ESCAPE": VirtualKeyShort.ESCAPE,
    "ESC": VirtualKeyShort.ESCAPE,
    "CONVERT": VirtualKeyShort.CONVERT,  # IME convert
    "SPACE": VirtualKeyShort.SPACE,
    "PRIOR": VirtualKeyShort.PRIOR,  # PAGE UP key
    "NEXT": VirtualKeyShort.NEXT,  # PAGE DOWN key
    "END": VirtualKeyShort.END,
    "HOME": VirtualKeyShort.HOME,
    "LEFT": VirtualKeyShort.LEFT,  # LEFT ARROW key
    "RIGHT": VirtualKeyShort.RIGHT,  # RIGHT ARROW key
    "UP": VirtualKeyShort.UP,  # UP ARROW key
    "DOWN": VirtualKeyShort.DOWN,  # DOWN ARROW key
    "SELECT": VirtualKeyShort.SELECT,
    "PRINT": VirtualKeyShort.PRINT,
    "EXECUTE": VirtualKeyShort.EXECUTE,
    "SNAPSHOT": VirtualKeyShort.SNAPSHOT,  # PRINT SCREEN key
    "INSERT": VirtualKeyShort.INSERT,
    "DELETE": VirtualKeyShort.DELETE,
    "HELP": VirtualKeyShort.HELP,
    "0": VirtualKeyShort.KEY_0,
    "1": VirtualKeyShort.KEY_1,
    "2": VirtualKeyShort.KEY_2,
    "3": VirtualKeyShort.KEY_3,
    "4": VirtualKeyShort.KEY_4,
    "5": VirtualKeyShort.KEY_5,
    "6": VirtualKeyShort.KEY_6,
    "7": VirtualKeyShort.KEY_7,
    "8": VirtualKeyShort.KEY_8,
    "9": VirtualKeyShort.KEY_9,
    "A": VirtualKeyShort.KEY_A,
    "B": VirtualKeyShort.KEY_B,
    "C": VirtualKeyShort.KEY_C,
    "D": VirtualKeyShort.KEY_D,
    "E": VirtualKeyShort.KEY_E,
    "F": VirtualKeyShort.KEY_F,
    "G": VirtualKeyShort.KEY_G,
    "H": VirtualKeyShort.KEY_H,
    "I": VirtualKeyShort.KEY_I,
    "J": VirtualKeyShort.KEY_J,
    "K": VirtualKeyShort.KEY_K,
    "L": VirtualKeyShort.KEY_L,
    "M": VirtualKeyShort.KEY_M,
    "N": VirtualKeyShort.KEY_N,
    "O": VirtualKeyShort.KEY_O,
    "P": VirtualKeyShort.KEY_P,
    "Q": VirtualKeyShort.KEY_Q,
    "R": VirtualKeyShort.KEY_R,
    "S": VirtualKeyShort.KEY_S,
    "T": VirtualKeyShort.KEY_T,
    "U": VirtualKeyShort.KEY_U,
    "V": VirtualKeyShort.KEY_V,
    "W": VirtualKeyShort.KEY_W,
    "X": VirtualKeyShort.KEY_X,
    "Y": VirtualKeyShort.KEY_Y,
    "Z": VirtualKeyShort.KEY_Z,
    "LWIN": VirtualKeyShort.LWIN,
    "RWIN": VirtualKeyShort.RWIN,
    "APPS": VirtualKeyShort.APPS,
    "SLEEP": VirtualKeyShort.SLEEP,
    "MULTIPLY": VirtualKeyShort.MULTIPLY,  # '*'
    "ADD": VirtualKeyShort.ADD,  # '+'
    "SEPARATOR": VirtualKeyShort.SEPARATOR,
    "SUBTRACT": VirtualKeyShort.SUBTRACT,
    "DECIMAL": VirtualKeyShort.DECIMAL,
    "DIVIDE": VirtualKeyShort.DIVIDE,
    "F1": VirtualKeyShort.F1,
    "F2": VirtualKeyShort.F2,
    "F3": VirtualKeyShort.F3,
    "F4": VirtualKeyShort.F4,
    "F5": VirtualKeyShort.F5,
    "F6": VirtualKeyShort.F6,
    "F7": VirtualKeyShort.F7,
    "F8": VirtualKeyShort.F8,
    "F9": VirtualKeyShort.F9,
    "F10": VirtualKeyShort.F10,
    "F11": VirtualKeyShort.F11,
    "F12": VirtualKeyShort.F12
}

I will extend the missing VirtualKeyShort bindings.

Afterwards you can use the KeyShorts by corresponding bindings

FlaUILibrary.Press Key t'ALT+SHIFT+SUBTRACT' <-- Should already work FlaUILibrary.Press Key t'ALT+SHIFT+ADD' <-- Should already work FlaUILibrary.Press Key t'ALT+SHIFT+EQUALS' <-- I don't find a corresponding from it

[Nepitwin]

If you want to test it simple checkout keyboard branch this contains the new application test by keyboard view.

Here you can see input which are received by keyboard or flaui.

Two ways to test :

Window events are triggered for key up and down event Input field can be used for keyboard input tests

[Mavnus04]

@Nepitwin I got the Keyboard branch my system and built. When I hit Alt on my keyboard, I get System While I hold Alt, and try to hit other keys, nothing else shows up in the 'Keyboard * Events'.

If I hold Shift first, then press other keys, they come up. Again, no more keys appear after Alt, if I do Shift then Alt.

Shift + - + Alt

As you can see, I am getting OemMinus for - key.

[Mavnus04]

Forgot to mention, that when I try ALT+SHIFT+SUBTRACT or ALT+SHIFT+ADD, it still doesn't do what it is supposed to.

[Nepitwin]

From sample application the intresting part is that WPF is sending System Event if you first execute ALT keyboard. If you first try to use STRG+ALT it displays the correct behvior.

If you execute first ALT by Down Event you got system. Your source code i think will handle to wait for ALT event :) So it does not recognize it. The question is why system key event is triggered.

[Nepitwin]

If your keyboard binding is first use ALT it will be executed as system by windows

https://social.msdn.microsoft.com/forums/vstudio/en-US/4355ab9a-9214-4fe1-87ea-b32dfc22946c/issue-with-alt-key-and-key-down-event?forum=wpf

https://docs.microsoft.com/en-us/dotnet/api/system.windows.input.keyeventargs.systemkey?redirectedfrom=MSDN&view=windowsdesktop-6.0#System_Windows_Input_KeyEventArgs_SystemKey

If your source code will handle on System and not Alt it probably work?

[Mavnus04]

Tried putting Shift before Alt by manually pressing keys and the application responded properly. Shift+Alt+- & Shift+Alt+= However, when I put it in the code, it still does NOT work. Pressing D as text is a sanity check for me that the application is receiving commands.

Show the info bar on the unity window 
    [Tags]  unity  
    Log to console  ...
    ${gamename} =  Set Variable  mygame
    Set Up Unity Game For Key Presses  ${gamename}

    FlaUILibrary.Focus  /Window[@Name='${gamename}']

    Log To Console  texts...
    FlaUILibrary.Press Key  t'D'
    Sleep  2s
    FlaUILibrary.Press Key  t'D'
    Sleep  2s

    Log To Console  shortcuts...
    FlaUILibrary.Press Key  s'SHIFT+ALT+SUBTRACT'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+ADD'
    Sleep  2s

[Nepitwin]

@Mavnus04

Oh cool you test by a unity based application.

Is their any kind of keyboard interaction be triggered by flaui?

For example you set a simple input by keyboard or an ESC to open the main menu?

If yes sounds good so far. If not i have to investigate which keyboard event is be triggered from flaui.

So it could be related for the tripple usage binding or probably that any keyboard events are not triggered by unity because the simulation from the keyboard is not handle correctly by windows :).

[Mavnus04]

@Nepitwin

Yes, it is a game created with Unity.

In my last example, I can send the character D and that works just fine, the game responds as it should. It is my sanity check that something is going through from FlaUI

But for some reason, combining those key presses will not go through. Shift, Alt, [Other Key], Doesn't seem to matter the order.

[Nepitwin]

@Mavnus04

Do you have a small source code sample application for it like the wpf example?

I could then analyze triggered events from unity and flaui to identify the issue probably.

[Mavnus04]

Sorry, I can't share anything from my company, but I could try making a simple Unity application which accepts and responds to some simple key presses. Give me a week or so since I have a lot of other tasks that I have to attend to.

[Nepitwin]

@Mavnus04

I have also thought of such an example application :).

No problem we have all work to do :)

[Mavnus04]

@Nepitwin Here is the unity application I created. Pressing certain keys will cause the background color to change.

W = White Y = Yellow Alt + Shift +

• Backspace = Blue
• Equals = Red
• Minus = Green
• Alpha0 = Grey

Application.zip

Manually pressing the keys, I have no issue changing the color. When I use FlaUI, only the single character key presses seem to work.

Change color of unity application
    [Tags]  unityapp
    Log to console  ...

    ${gamename} =  Set Variable  Keypress
    Attach application by name  ${gamename}
    FlaUILibrary.Focus  /Window[@Name='${gamename}']

    Log To Console  texts...
    FlaUILibrary.Press Key  t'Y'
    Sleep  2s
    FlaUILibrary.Press Key  t'W'
    Sleep  2s

    Log To Console  shortcuts...
    FlaUILibrary.Press Key  s'SHIFT+ALT+BACK'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+ADD'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+SUBTRACT'
    Sleep  2s

The source code is 55MB, so I am not sure how to share that with you, if you need it? GitHub said the upload limit is 25MB

[Nepitwin]

@Mavnus04

Thanks a lot i will investigate what happens on flaui side now by this sample application and hopefully find the issue 👍

[Nepitwin]

What i can see by my tests that an array is created with correct converted syntax

https://github.com/FlaUI/FlaUI/blob/58b5954349afe2343aeb16ea55b19962b28cd3e1/src/FlaUI.Core/WindowsAPI/Constants.cs#L934

SHIFT (0x10) + ALT (0x12) + BACKSPACE (0x8) [16, 18, 8]

Type simualtenously method is called which shoud execute all these keyboard bindings

https://github.com/FlaUI/FlaUI/blob/6841da31e13d74f8ad2128e6a48cfa00c9e3501c/src/FlaUI.Core/Input/Keyboard.cs#L110

It's quite strange i will try to implement a direct test in C# usage without wrapper libary to avoid possible data type converting issues. If the C# solution the issue is not on FlaUI it's on wrapper data type converting by robotframework-flaui wrapper which has to be identified why commands are not working like expected.

[Nepitwin]

@Mavnus04

Okay i found some issues related to FlaUI by Alt Keyboard usage.

https://github.com/FlaUI/FlaUI/issues/295 https://github.com/FlaUI/FlaUI/issues/320

Also the C# sample code have the same behavior like robotframework usage here.

using FlaUI.UIA3;
using FlaUI.Core.Input;
using FlaUI.Core.WindowsAPI;

var app = FlaUI.Core.Application.Launch("F:\\Unity\\Keypress.exe");
using (var automation = new UIA3Automation())
{
    var window = app.GetMainWindow(automation);
    Thread.Sleep(6000);
    Keyboard.TypeSimultaneously(VirtualKeyShort.KEY_Y);
    Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.ALT, VirtualKeyShort.BACK);
    Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.LCONTROL, VirtualKeyShort.BACK);
    Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.CONTROL, VirtualKeyShort.BACK);

    Keyboard.Type(VirtualKeyShort.SHIFT);
    Keyboard.Type(VirtualKeyShort.CONTROL);
    Keyboard.Type(VirtualKeyShort.BACK);
}

I will retry FlaUI by a master build from current source code if it's working then i will update the flaui wrapper to the latest master version. But it seems a known issue on flaui that the combinations usage by ALT does not work like expectly.

[Nepitwin]

From my point of view it's a keyboard issue on flaui. So the general idea was is to replace the keyword module by something different by keyword modules like

Pyinput - https://nitratine.net/blog/post/simulate-keypresses-in-python/ Keyboard - https://pypi.org/project/keyboard/ Pyautogui - https://github.com/asweigart/pyautogui

But after some additional analyze i also found https://stackoverflow.com/questions/13564851/how-to-generate-keyboard-events

Which cause issues on keyboard handling by ui automation especially for games.

So in general my plan is now to implement a plugin system like selenium already has.

https://github.com/robotframework/SeleniumLibrary/blob/master/docs/extending/extending.rst#Plugins

This interface could be used for use cases like yours to implement workaround keywords by handling seperatly on your use case. I will implement as soon as possible to make a new release and support this feature. So afterwards this ui automation library can be used and extended by anyone with this plugin system for all specific acceptance criterias.

[Mavnus04]

Nice investigation! Looking forward to the new version. Thank you very much.

[Mavnus04]

@Nepitwin How is it going? Wondering if you could provide an update? Thanks.

[Nepitwin]

@Mavnus04

Currently i'm busy at work to do a quick fix for it and i think about the plugin mechanis if this behavior should fix it. I'm not sure. But you can already try a second solution to write a simple Keyword by your own in python code.

I think you know that this possiblity is always supported by robotframework.

Simple write a Python file and define a custom keyword which probably contains a sub call by another keyboard library which you include and use for your use case.

It would not solve the issue on flaui side but for thie one case your e2e test would work :).

[Mavnus04]

I already have something that can send all these key commands for my testing purposes. My hope was to replace it with FlaUI since it supports so much more.

Please let me know once you have a fix in, thanks.

[Mavnus04]

@Nepitwin Hey, checking for an update?

[Nepitwin]

Currently no news about an upcoming change how to fix your issue by unity. The only thing what i found is that flaui is sending by SendInput controls keyboard interactions. https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput