Virtual key input is blocked in Windows

[chi3236]

We used WM_KEYINPUT function and keybd_event funtion to drive the truck automatically, but thevirtual key input with those functions is blocked. Specifically, virtual key input is working on other programs, such as notepad or ms word, but we could not handle the truck with virtual key input.

[zappybiby]

The fix for this is to replace the deprecated keybd_event with SendInput as described here: http://stackoverflow.com/a/41057051

I figured this out after looking at http://stackoverflow.com/a/40318692

[zappybiby]

Possibly something like this? https://github.com/ajchili/ChosunTruck/commit/6f3dadcef61041d8729bd8443feee7b4227a4796#diff-3644f6bac665e291284f6df8b749eda1R205

to be honest, I don't know what I'm doing... I tried though!! 👶

[zappybiby]

Update: Input w/ SendInput not working with my current implementation, however I still believe it is what is the issue here. https://github.com/ajchili/ChosunTruck/commit/5ab03b401a5a8acaaa37059496034631ac371491

Also, for the past couple hours now I have been trying to get GPUMat to work... keep getting error

OpenCV Error: The function/feature is not implemented You should explicitly call download method for cuda::GpuMat object...

[chi3236]

I tried umat instead of mat for using gpu before, but there was an incompatibility problem. I gonna try it again

[chi3236]

http://stackoverflow.com/questions/35138778/sending-keys-to-a-directx-game

By this Q&A, SendMessage function is not moking actual key input, and DirectX app cannot be controled by this function. Keybd_Input can be used for DirectX app but unfortunately it seems like that function is blokced by ETS2 client. We should try other methods like direct input hooking...

[zappybiby]

Maybe use postmessage instead of sendmessage?

From further reading on stackoverflow, SendInput should definately work. I can check using Spy++ when I get home to confirm this.

I'm thinking that using hardware scan codes might work, like they did here: http://stackoverflow.com/a/18854441

[zappybiby]

https://github.com/ajchili/ChosunTruck/commit/b4ac41f21e477142bfe61ed2e847eefb407e86f1 This is my attempt at fixing input. Although I can't check to see if it works because the program outputs "go straight" once and then crashes :(

Edit: Also worth noting that we could try

ip.ki.wVk = 0; //We're doing scan codes instead
    ip.ki.dwExtraInfo = 0;

    //This let's you do a hardware scan instead of a virtual keypress
    ip.ki.dwFlags = KEYEVENTF_SCANCODE;
    ip.ki.wScan = 0x1E;  //Set a unicode character to use (A)

to use hardware scan codes instead of virtual key presses...

[zappybiby]

After more testing (https://github.com/ajchili/ChosunTruck/commit/750a79d76623a28676997cf8f9587e97296515c6), I found that the program freezes after reading line 161 (cout << "go left ";). Unfortunately can't debug because it doesn't crash, just freezes after first "go straight" output...

[chi3236]

What does "freeze" mean? Does program stop without crash or something? Maybe that is because of infinite loop without call stack push :(

[zappybiby]

Yeah, program freezes because of the issue described here: https://github.com/bethesirius/ChosunTruck/issues/6

Freezing means that the program stops working, but doesn't exit or return with an exception. It just hangs and eventually needs to be closed with task manager. I ended up using"-nudge" with Dr. Memory to give me some info on the problem. However I can't get it to display any useful stack trace...

In a recent commit (https://github.com/ajchili/ChosunTruck/commit/6c96498f79928c228d8f51e9213962d374351224), I edited the code to use hardware scan input, I'm really hoping that works. If I can fix this buffer overflow error, I think we have a good chance of having a working Windows version! Otherwise, we will need to use DirectInput

[zappybiby]

Still stuck on the buffer overflow error, but in the meantime I changed the input to follow the original SendMessage format you created. Basically, it goes in the original order that you created. I did change the "0x74" (F5) to "0x44" ('D'), however. I assume that was a typo, since F5 is not really relevant in changing directions.

See https://github.com/ajchili/ChosunTruck/commit/bced2f5e6682acd9bca9fc5d2ea9a181c0a348f0?diff=unified#diff-3644f6bac665e291284f6df8b749eda1

[zappybiby]

New Developments! I used Spy++ to see what inputs were being taken by ETS2.

Also, I used PE explorer to find out if ETS2 is importing any DirectX dll's. It is NOT using any DirectX dll's, and is instead using Win32 API. Which is great, since it means we will not have to account for DirectInput.

This means that keybd_event, SendMessage, and/or PostMessage should work when sending key inputs. I confirmed this by downloading the USER32.dll (http://pastebin.com/FbBEZyUn)

Interestingly enough, SendMessage is not in there, only SendMessageW is shown. 'W' stands for Wide or Unicode.

Here is the Spy++ log when physically pressing the keys on my keyboard:

Real Keys

<000001> 001208CA P WM_KEYDOWN nVirtKey:'A' cRepeat:1 ScanCode:1E fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000002> 001208CA P WM_CHAR chCharCode:'97' (97) cRepeat:1 ScanCode:1E fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000003> 001208CA P WM_KEYUP nVirtKey:'A' cRepeat:1 ScanCode:1E fExtended:0 fAltDown:0 fRepeat:1 fUp:1

<000004> 001208CA P WM_KEYDOWN nVirtKey:'S' cRepeat:1 ScanCode:1F fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000005> 001208CA P WM_CHAR chCharCode:'115' (115) cRepeat:1 ScanCode:1F fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000006> 001208CA P WM_KEYUP nVirtKey:'S' cRepeat:1 ScanCode:1F fExtended:0 fAltDown:0 fRepeat:1 fUp:1

<000007> 001208CA P WM_KEYDOWN nVirtKey:'D' cRepeat:1 ScanCode:20 fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000008> 001208CA P WM_CHAR chCharCode:'100' (100) cRepeat:1 ScanCode:20 fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000009> 001208CA P WM_KEYUP nVirtKey:'D' cRepeat:1 ScanCode:20 fExtended:0 fAltDown:0 fRepeat:1 fUp:1

<000010> 001208CA P WM_KEYDOWN nVirtKey:'W' cRepeat:1 ScanCode:11 fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000011> 001208CA P WM_CHAR chCharCode:'119' (119) cRepeat:1 ScanCode:11 fExtended:0 fAltDown:0 fRepeat:0 fUp:0

<000012> 001208CA P WM_KEYUP nVirtKey:'W' cRepeat:1 ScanCode:11 fExtended:0 fAltDown:0 fRepeat:1 fUp:1

[zappybiby]

Looking at the Spy++ log, it is evident that we should be using PostMessage to send input. (See the 'P')

[zappybiby]

Switched to PostMessage https://github.com/ajchili/ChosunTruck/commit/e1bc37e60ae869997b9847ff08f958191c7d19dc

Spy++ shows that it is giving nearly identical presses when comparing real and simulated keys. I am not sure why this isn't working. Should probably increase Sleep() time, keys are being pressed far too quickly.

Simulated Keys

<003546> 001208CA P WM_KEYDOWN nVirtKey:'D' cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0
<003547> 001208CA P WM_CHAR chCharCode:'100' (100) cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0
<003548> 001208CA P WM_KEYUP nVirtKey:'D' cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0
<003549> 001208CA P WM_CHAR chCharCode:'100' (100) cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0
<003550> 001208CA P WM_KEYUP nVirtKey:'A' cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0
<003551> 001208CA P WM_CHAR chCharCode:'97' (97) cRepeat:0 ScanCode:00 fExtended:0 fAltDown:0 fRepeat:0 fUp:0

[chi3236]

Sorry for my late reply :( . I am studying ML with Dr. Ng's lecture and working on object detection with Google Tensorflow. By the way, I pulled e1bc37e to my workstation, and I think auto steering will work if key press problem is fixed. Thank you for your interest and I will apply object detection to my project soon. Also, linux branch and master branch will be merged in several days.

[zappybiby]

No problem!

You may want to make the SendInput an array. I've tried using both Scan codes and Virtual keys but it's still not working for me. Frustrating! I'm still not sure if the KEYEVENTF_SCANCODE flag is actually defined...

For example:

INPUT ip[2];
ip[0].type = INPUT_KEYBOARD;
ip[0].ki.time = 0;
ip[0].ki.wVk = 0; // scan codes
ip[0].ki.dwExtraInfo = 0;
// Releases 'D'
ip[0].ki.wScan = 0x44;
ip[0].ki.dwFlags = KEYEVENTF_KEYUP;

 // Releases 'A'
ip[1].ki.wScan = 0x41;
ip[1].ki.dwFlags = KEYEVENTF_KEYUP;
SendInput(2, &ip, sizeof(ip)); // Note: '&' probably not needed, already a pointer.
Sleep(100);

I think we will need to use SendInput because PostMessage does not alter LPARAM as far as I know. Also, using SendInput makes my program crash. PostMessage has great FPS.

[zappybiby]

Input works now: https://github.com/ajchili/ChosunTruck/commit/b8746dfb5a8bf8b8274f378fb361b50cd27731e9

Thanks to Komat for suggesting to change the di8.keyboard/fusion.mouse in the controls.sii to sys.keyboard/sys.mouse.

Edit: I mean it works in that ETS2 is receiving the input and moving the truck. However, it is sending keys too fast, and adding Sleep() makes the program crash... So for now it's not working "correctly"

Edit2: Now with sys.mouse, we can change SendInput to use mouse instead, which would mimic input in linux branch.

Edit3: https://github.com/ajchili/ChosunTruck/commit/efcd6139342a6a9c842b425b1c8350283990975f zeromemory is kind of redundant so getting rid of that.

[zappybiby]

We now know to use SetCursorPos to control the mouse. I think this issue can be closed now?

[chi3236]

OK. Input in windows is now working. Thanks!

[MatthewBerk]

So I know this thread is three years old, though it helped me better understand win32api.keybd_event from Python's pywin32 package, and I hope my discoveries could help others who happen upon this thread.

I was trying to figure out the purpose of the scan code argument since I found putting 0 instead of the actual scan code of the key worked for applications like notepad and games like 7 days to die which seem to use DirectX. I came upon this thread and decided to try out the method on Euro Truck Simulator 2 and it lead to some interesting discoveries.

To keep it short:

i = "a"
win32api.keybd_event(0, win32api.MapVirtualKey(VK_CODE[i], 0), 0, 0)
# Works for some DirectX games like Euro Truck Simulator 2 but not other applications like notepad.
# Doesn't work for games like 7 days to die which is reported to use DirectX. 
sleep(3.05)
win32api.keybd_event(0, win32api.MapVirtualKey(VK_CODE[i], 0), win32con.KEYEVENTF_KEYUP, 0)

i = "w"
win32api.keybd_event(VK_CODE[i], win32api.MapVirtualKey(VK_CODE[i], 0), 0, 0)
# Works for some DirectX games like Euro Truck Simulator 2  and other applications like notepad.
# Works for games like 7 days to die and Stardew Valley.
sleep(3.05)
win32api.keybd_event(VK_CODE[i], win32api.MapVirtualKey(VK_CODE[i], 0), win32con.KEYEVENTF_KEYUP, 0)

i = "d"
win32api.keybd_event(VK_CODE[i], 0, 0,0) 
# Doesn't work for some DirectX games like Euro Truck Simulator 2  BUT works for other applications like notepad.
# Works for games like 7 days to die and Stardew Valley.
sleep(3.05)
win32api.keybd_event(VK_CODE[i], 0, win32con.KEYEVENTF_KEYUP, 0)

So, so far it seems to get the best results from win32api.keybd_event, should specify virtual key code and scan code 'win32api.keybd_event(VK_CODE[i], win32api.MapVirtualKey(VK_CODE[i], 0), 0, 0)' Some games take vk code, some only accept scan code. So far have not found a downside to sending both.

I don't mean no downside to keybd_event, since an argument can be made due to its deprecation and documentation suggests using SendInput. Though in python at least, SendInput isn't in win32api but in a different package called ctypes ctypes.windll.user32.SendInput. Probably different in c++ but I am working with Python for now.

Note: I have only tested this on one system, so unsure if get different results using different systems.