Replaying events doesn't work for some types of trackpad gestures

calftrail / Touch

(Multitouch actually) My reverse engineering work of OS X gesture and multitouch events that led to Sesamouse (GPL v2.0 license)

http://calftrail.com/blog/

136 stars 19 forks source link

Replaying events doesn't work for some types of trackpad gestures #3

Open noah-nuebling opened 4 years ago

noah-nuebling commented 4 years ago

I'm trying to record touch gestures using TouchRecorder and then replay them using TouchExtractor. This doesn't seem to work in all cases. Here are my findings so far:

These types of events seem to work flawlessly when replaying

Two finger pinch (magnification) events
Rotation events
Double tap (smart zoom) events

There seem to be some issues with the following types of events

Replaying two finger swipe (Scrolling & Swipe between pages) events doesn't seem to work in general, with one exception – When you physically start swiping between pages (Two-finger horizontal swipe in Safari, Xcode, etc.) before the replay starts – then the replay will cause the page to move.
Four finger pinch (Launchpad & Show Desktop) replays don't seem to work at all.
Force touch (Look up & data detectors) replays don't seem to work at all.
Three/four finger swipe (Mission Control, etc.) event replays don't seem to work at all.

noah-nuebling commented 4 years ago

My goal is to translate mouse movement into 3 finger swipes for my little app Mac Mouse Fix as this would allow users to use Spaces and Mission Control from their mouse in a really responsive way.

The fact that replays don't work for 3 finger swipes, makes me think that additional reverse engineering work might have to be done to make the touch synthesis work properly with newer versions of macOS. I'd be willing to give that a try but I have no reverse engineering experience and it looks quite intimidating.

If anyone has more knowledge or advice on how to achieve 3 finger swipe synthesis on newer macOS versions I'd be super glad if you let me know!

noah-nuebling commented 4 years ago

I found a solution!

In TouchExtractorAppDelegate - replayEvents: the line

if (CGEventGetType(e) == 29) {

needs to be changed to

if (CGEventGetType(e) == 29 || CGEventGetType(e) == 30) {

for the replay to work.

So it seems that the type 30 events weren't a thing / weren't used this way when this library was written. They might also be the key to simulating gestures!

One more thing to note if you want to play with this yourself is that you have to uncheck and recheck the checkbox at System Preferences -> Security & Privacy -> Privacy -> Accessibility -> TouchExtractor whenever you rebuild after making any change to the code.