search

ogallagher / terry

Terry the virtual secreTERRY
0 stars 0 forks source link

Create Watcher for system keyboard and mouse input #20

Open ogallagher opened 4 years ago

ogallagher commented 4 years ago

In addition to being used for demonstration learning, the watcher will also be useful for enabling Terry to respond to input events even when not visible. For example, if I want Terry to wake on a key sequence (like cmd/ctrl + shift + t), the watcher will be necessary.

ogallagher commented 4 years ago

The jnativehook library uses bindings to C programs through to create hooks for system events. I will use these classes in my watcher.

ogallagher commented 4 years ago

I’m currently integrating jnativehook into my Watcher, but am having issues handling key events.

ogallagher commented 4 years ago

As of today I feel I’ve done all I can to try and get jnativehook to work, but for some reason the key event problem persists for no apparent reason. I’ve even gone ahead and acquired libuiohook separately (the c library on which jnativehook is based) and compiled its demo on my laptop to prove that it can indeed process all the key events I’m looking for. But for some reason those events do not get passed on to the java classes in jnativehook.

I’ve also tried downloading jnativehook, changing the source (both java and JNI c) to insert debugging comments and see where the events are lost, but with no luck. Even after finally managing to recompile the jnativehook jar library and use it in a test program my changes to the JNI c code don’t seem to carry through... I’ve hit a wall here for now.

ogallagher commented 4 years ago

A few days ago I confirmed that the watcher code using jnativehook works perfectly in windows, so I’m moving ahead assuming I’ll figure out what isn’t working on mac eventually.

Today I drafted demonstration learning, which involved creating a lesson that triggers the watcher to record a new demonstration, which compiles into a new action on completion. The new action is a sequence of states (mouse and keyboard manipulations) with prescribed args according to each state’s requirements.