Deinhofer/jnativehook mac catalina fix

deinhofer commented 4 years ago

This PR fixes the issue #328 by upgrading to jnativehook branch 2.1.

It was successfully tested on Mac OS Catalina, Mac OS 10.11, Ubuntu 19.04 and Raspberry Pi 4. Windows still has to be tested extensively.

@ChrisVeigl , @klues : Could you do the follwing tests?

1.

git checkout -b deinhofer/jnativehook-mac-catalina-fix https://github.com/asterics/AsTeRICS.git
cd AsTeRICS/ARE
ant clean
ant run-debug

Then test models: 

bin/ARE/models/componentTests/actuators/Keyboard_test_all_chars.acs (Open editor and click into it. After 5 seconds some text should be written into the editor.)
bin/ARE/models/componentTests/sensors/KeyCapture.acs
bin/ARE/models/componentTest/sensors/MouseCapture_test.acs

2. Do a long-run test. Start the ARE and keep it running during the night. Does anything freeze?

ChrisVeigl commented 4 years ago

i can confirm the same test results as Klues reported. (tested on Win10)

"umlaute" are not printed by the keyboard plugin
the "block" function of the KeyCapture plugin does not seem to work
the "block" function of the MouseCapture plugin switches to relative coordinate output (which is AFAIK the intended behaviour)

deinhofer commented 4 years ago

Here is my result for Keyboard_test_all_chars.acs:

abcdefghijklmnopqrstuvwxzy
,;.:ß?ä 
123456-ü+0
ABCDEFGHIJKLMNOPQRSTUVWXZY
!" $ &_Ü* '
Umlaute:     #

So the a is correct. Unfortunately jnativehook only supports an english keyboard layout but nevertheless the a should be correct.

I think blocking should actually work on Windows but not on Linux due to limitations on X11, see this.

The main purpose of the test was to check if capturing and emulation generally works and the ARE does not freeze.

Please also perform test 2 (long term test). Finally, perform test 1 again using the master branch to compare the results. Call ant clean && ant run to ensure using the correct binary.

Here is my result for Keyboard_test_all_chars.acs (master branch):

abcdefghijklmnopqrstuvwxzy
,;.:ß?ä 
123456-ü+0
ABCDEFGHIJKLMNOPQRSTUVWXZY
!" $ &_Ü* '
Umlaute:     #

klues commented 4 years ago

did some kind of long term test (about 6 hours), nothing crashed, I only got the following exception:

2020-03-12 15:56:59.628 SEVERE [Main$1 uncaughtException] in Thread <AWT-EventQueue-0>: component must be showing on the screen to determine its location
     [java] java.awt.IllegalComponentStateException: component must be showing on the screen to determine its location
     [java]     at java.awt.Component.getLocationOnScreen_NoTreeLock(Component.java:2062)
     [java]     at java.awt.Component.getLocationOnScreen(Component.java:2036)
     [java]     at javax.swing.text.JTextComponent$InputMethodRequestsHandler.getTextLocation(JTextComponent.java:4643)

but doesn't seem that this is related to jnativehook. I've tested the test model KeyCapture.acs and it still works after running 6 hours.

Did also test the Keyboard_test_all_chars.acs on the master branch, this is the output:

abcdefghijklmnopqrstuvwxzy
,;.:ß?ä 
123456-ü+0
ABCDEFGHIJKLMNOPQRSTUVWXZY
!" $ &_Ü* '
Umlaute:     #

seems to be the same as yours, but different from my test on the deinhofer/jnativehook-mac-catalina-fix branch.

deinhofer commented 4 years ago

did the tests as requested:

* `Keyboard_test_all_chars.acs` in texteditor the following was written:
  ```
  âbcdefghijklmnopqrstuvwxyz
  ,;.:-_ä 
  123456#ß´0
  ABCDEFGHIJKLMNOPQRSTUVWXYZ
  !" $ &'?`°
  Umlaute:     
  ```

  does not seems to be all correct (no umlaute, wrong numbers above 6, first `a` letter with accent.

* `KeyCapture.acs` generally works, but "Block" and "Unblock" do not have any effect (don't know if it should)

* `MouseCapture_test.acs` works generally, "PollMouse" has no effect and on "Block" the absolute values shown in the graph are jumping to a different level of absolute values.

did the tests as requested:

* `Keyboard_test_all_chars.acs` in texteditor the following was written:
  ```
  âbcdefghijklmnopqrstuvwxyz
  ,;.:-_ä 
  123456#ß´0
  ABCDEFGHIJKLMNOPQRSTUVWXYZ
  !" $ &'?`°
  Umlaute:     
  ```

  does not seems to be all correct (no umlaute, wrong numbers above 6, first `a` letter with accent.

* `KeyCapture.acs` generally works, but "Block" and "Unblock" do not have any effect (don't know if it should)

* `MouseCapture_test.acs` works generally, "PollMouse" has no effect and on "Block" the absolute values shown in the graph are jumping to a different level of absolute values.

Could you please test Keyboard_test_all_chars.acs again on Windows? In my Win10 VM I get a correct a.

klues commented 4 years ago

Got exactly the same result as before with â. But since nothing changed within this branch since the last test, this doesn't surprise me ;)

No Exception in the console.

deinhofer commented 4 years ago

Got exactly the same result as before with â. But since nothing changed within this branch since the last test, this doesn't surprise me ;)

No Exception in the console.

Yeah but as I said, on my Win10 (both Virutal box and dual boot) the a is correct. I have tested it with english and german keyboard layout. Always ok.

Could you download the zip below, extract it and copy the jar to the bin\ARE folder, replacing the other jar? And do the test again? Could you also post your keyboard layout and win10 update nr? cp asterics.mw.jnativehook-mvn.jar bin\ARE\asterics.mw.jnativehook.jar

asterics.mw.jnativehook-mvn.zip

@bmedicke Please also perform the test on your Mac again with the new jar.

deinhofer commented 4 years ago

i can confirm the same test results as Klues reported. (tested on Win10)

"umlaute" are not printed by the keyboard plugin

the "block" function of the KeyCapture plugin does not seem to work

the "block" function of the MouseCapture plugin switches to relative coordinate output (which is AFAIK the intended behaviour)

In your tests, did you have a a or a â??

klues commented 4 years ago

I replaced the .jar file and rerun the test.

The result is the following, Umlaute are still missing, but the normal letters seem to be correct:

abcdefghijklmnopqrstuvwxyz
,;.:-_ä 
123456#ß´0
ABCDEFGHIJKLMNOPQRSTUVWXYZ
!" $ &'? °
Umlaute:

In the console I get the following errors while running the test:

[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04070407!
[java]
[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04090409!
[java]
[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04090436!

The same errors are occuring if I just run ARE with the autorun.acs and then press any key on my keyboard anywhere (e.g. within the Browser).

My keyboard Layout/System locale is "Deutsch (Österreich) - Deutsch Tastatur", but I have also these for selecting in the menu: http://prntscr.com/s54ehj Win 10 Version is: Windows 10 Education, Version 1803, Build 17134.1425

deinhofer commented 4 years ago

I replaced the .jar file and rerun the test.

The result is the following, Umlaute are still missing, but the normal letters seem to be correct:
abcdefghijklmnopqrstuvwxyz
,;.:-_ä 
123456#ß´0
ABCDEFGHIJKLMNOPQRSTUVWXYZ
!" $ &'? °
Umlaute:     
In the console I get the following errors while running the test:
[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04070407!
[java]
[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04090409!
[java]
[java] Apr 24, 2020 11:09:21 AM org.jnativehook.GlobalScreen$NativeHookThread enable
[java] SCHWERWIEGEND: refresh_locale_list [628]: Could not find keyboard map for locale 0X04090436!
The same errors are occuring if I just run ARE with the autorun.acs and then press any key on my keyboard anywhere (e.g. within the Browser).

My keyboard Layout/System locale is "Deutsch (Österreich) - Deutsch Tastatur", but I have also these for selecting in the menu: http://prntscr.com/s54ehj Win 10 Version is: Windows 10 Education, Version 1803, Build 17134.1425

Does key capturing still work? e.g. pressing F8 for opening a model?

klues commented 4 years ago

yes, key capturing and F-Key-functionalities are working.

bmedicke commented 4 years ago

ABC layout

abcdefghijklmnopqrstuvwxyz
,,..—‘ 
123456/[]0
abcdefghijklmnopqrstuvwxyz
12 4 6/[] \
umlaute.     \

U.S. International - PC

abcdefghijklmnopqrstuvwxyz
,,..—‘
123456/[]0
abcdefghijklmnopqrstuvwxyz
12 4 6/[] \
umlaute.     \

AT layout

abcdefghijklmnopqrstuvwxzy
,,..ßßä 
123456-ü+0
abcdefghijklmnopqrstuvwxzy
12 4 6-ü+ #
umlaute.     #

Looks like it's not triggering shift.

ant run-debug 2>&1 > run-debug.log

deinhofer commented 4 years ago

ABC layout

abcdefghijklmnopqrstuvwxyz
,,..—‘ 
123456/[]0
abcdefghijklmnopqrstuvwxyz
12 4 6/[] \
umlaute.     \

U.S. International - PC

abcdefghijklmnopqrstuvwxyz
,,..—‘
123456/[]0
abcdefghijklmnopqrstuvwxyz
12 4 6/[] \
umlaute.     \

AT layout

abcdefghijklmnopqrstuvwxzy
,,..ßßä 
123456-ü+0
abcdefghijklmnopqrstuvwxzy
12 4 6-ü+ #
umlaute.     #

Looks like it's not triggering shift.

ant run-debug 2>&1 > run-debug.log

Ok, so no freezing on Mac OS? Does capturing (e.g. F5 or F8) still work?

bmedicke commented 4 years ago

Ok, so no freezing on Mac OS? Does capturing (e.g. F5 or F8) still work?

No freezing, F5/F8 work and so do KeyCapture.acs and MouseCapture_test.acs.

deinhofer commented 4 years ago

I will leave it with the currently checked-in jar, as it's gonna be annoying to always have the error message. The á is not that critical as it only happens on some windows systems and on Windows jnativehook based keyboard emulation is not the default.

asterics / AsTeRICS

Deinhofer/jnativehook mac catalina fix #329