Abort at launch on macOS 10.15

[teddywing]

I’m getting the following error when attempting to run Warpd:

$ ./bin/warpd -f
Starting warpd vv1.3.4-osx (built from: 567205b) ()
Abort trap: 6

The same happens when installing and running via Launchd:

$ launchctl list | grep warpd
-   -6  com.warpd.warpd

Do you have any guidance for how to debug the problem?

I’m running macOS 10.15:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.1
BuildVersion:   19B88

This happens both on the latest master and on v1.3.4:

$ git log --oneline -1
567205b (HEAD -> master, origin/master, origin/HEAD) macos: Wait for accessibility permissions

The binary seems to build correctly:

$ make
cc -c src/config.c -o src/config.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/grid.c -o src/grid.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/grid_drw.c -o src/grid_drw.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/hint.c -o src/hint.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/histfile.c -o src/histfile.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/history.c -o src/history.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/input.c -o src/input.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/mouse.c -o src/mouse.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/normal.c -o src/normal.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/screen.c -o src/screen.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/scroll.c -o src/scroll.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
cc -c src/warpd.c -o src/warpd.o -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2
src/warpd.c:170:45: warning: suggest braces around initialization of subobject [-Wmissing-braces]
        struct input_event activation_events[8] = {0};
                                                   ^
                                                   {}
1 warning generated.
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/mouse.o src/platform/macos/mouse.m
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/screen.o src/platform/macos/screen.m
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/input.o src/platform/macos/input.m
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/hint.o src/platform/macos/hint.m
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/macos.o src/platform/macos/macos.m
cc -g -Wall -Wextra -pedantic -Wno-deprecated-declarations -Wno-unused-parameter -std=c99 -DVERSION='"v1.3.4-osx (built from: 567205b)"' -DCOMMIT=\"\" -D_DEFAULT_SOURCE -D_FORTIFY_SOURCE=2     -c -o src/platform/macos/window.o src/platform/macos/window.m
mkdir bin
cc -o bin/warpd src/config.o src/grid.o src/grid_drw.o src/hint.o src/histfile.o src/history.o src/input.o src/mouse.o src/normal.o src/screen.o src/scroll.o src/warpd.o src/platform/macos/mouse.o src/platform/macos/screen.o src/platform/macos/input.o src/platform/macos/hint.o src/platform/macos/macos.o src/platform/macos/window.o -framework cocoa -framework carbon

$ otool -L bin/warpd
bin/warpd:
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 162.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
    /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1875.0.0)
    /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 52.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1670.10.0)
    /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1348.4.0)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1065.2.0)
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1670.10.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

$ lldb ./bin/warpd
(lldb) target create "./bin/warpd"
Current executable set to './bin/warpd' (x86_64).
(lldb) process launch --stop-at-entry -- -f
Process 24399 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0x0000000100232000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100232000 <+0>: popq   %rdi
    0x100232001 <+1>: pushq  $0x0
    0x100232003 <+3>: movq   %rsp, %rbp
    0x100232006 <+6>: andq   $-0x10, %rsp
Target 0: (warpd) stopped.
Process 24399 launched: '.../rvaiya--warpd/bin/warpd' (x86_64)
(lldb) c
Process 24399 resuming
Starting warpd vv1.3.4-osx (built from: 567205b) ()
Waiting for accessibility permissions
Accessibility permission granted, proceeding
Process 24399 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff6eb1049a libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff6eb1049a <+10>: jae    0x7fff6eb104a4            ; <+20>
    0x7fff6eb1049c <+12>: movq   %rax, %rdi
    0x7fff6eb1049f <+15>: jmp    0x7fff6eb0a3b7            ; cerror_nocancel
    0x7fff6eb104a4 <+20>: retq
Target 0: (warpd) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff6eb1049a libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff6ebcd6cb libsystem_pthread.dylib`pthread_kill + 384
    frame #2: 0x00007fff6ea98a1c libsystem_c.dylib`abort + 120
    frame #3: 0x00007fff362fe907 HIToolbox`islGetInputSourceListWithAdditions.cold.1 + 23
    frame #4: 0x00007fff36071778 HIToolbox`islGetInputSourceListWithAdditions + 1841
    frame #5: 0x00007fff3607101f HIToolbox`TSMGetInputSourceCountWithFilteredAdditions + 35
    frame #6: 0x00007fff3606fcfd HIToolbox`TISCreateInputSourceList + 84
    frame #7: 0x00007fff36076c37 HIToolbox`CopyInputSourceRefForInputSourceID + 89
    frame #8: 0x00007fff362668c6 HIToolbox`TISSetGestureHandwritingState + 29
    frame #9: 0x00007fff3606f978 HIToolbox`SyncHandwritingHotKey + 46
    frame #10: 0x00007fff3606ea4c HIToolbox`_FirstEventTime + 876
    frame #11: 0x00007fff3606e58a HIToolbox`RunCurrentEventLoopInMode + 49
    frame #12: 0x00007fff3606e2c9 HIToolbox`ReceiveNextEventCommon + 356
    frame #13: 0x00007fff3606e147 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff346f3864 AppKit`_DPSNextEvent + 990
    frame #15: 0x00007fff346f25d4 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff346ecd76 AppKit`-[NSApplication run] + 658
    frame #17: 0x0000000100009f1b warpd`osx_run(loop=(warpd`daemon_loop at warpd.c:164)) at macos.m:183:2
    frame #18: 0x0000000100006774 warpd`main(argc=2, argv=0x00007ffeefbff3d8) at warpd.c:448:3
    frame #19: 0x00007fff6e9c12e5 libdyld.dylib`start + 1
    frame #20: 0x00007fff6e9c12e5 libdyld.dylib`start + 1

[rvaiya]

Thanks for the detailed debug info. I can confirm 567205b works on Monterey (support for older versions might be a bit flaky). My guess is that this is an accessibility issue. Were you prompted to add your terminal to the accessibility settings when you first ran the binary?

Your accessibility settings should look something like this:

[teddywing]

I’m not prompted to allow Accessibility access when I run the binary normally, but I was prompted when running it through lldb. This appeared to be acknowledged:

(lldb) c
Process 24399 resuming
Starting warpd vv1.3.4-osx (built from: 567205b) ()
Waiting for accessibility permissions
Accessibility permission granted, proceeding

Both my terminal (iTerm) and the warpd binary are listed and checked in System Preferences > Security & Privacy > Privacy > Accessibility. I can confirm this in the TCC database:

$ sudo cp /Library/Application\ Support/com.apple.TCC/TCC.db /tmp/
$ sudo sqlite3 /tmp/TCC.db
...
sqlite> SELECT service, client, allowed FROM access WHERE service = 'kTCCServiceAccessibility' AND allowed = 1 AND (client LIKE '%warpd%' OR client LIKE '%iterm2');
service                    client                              allowed
-------------------------  ----------------------------------  -------
kTCCServiceAccessibility   <path/to>/rvaiya--warpd/bin/warpd   1
kTCCServiceAccessibility   com.googlecode.iterm2               1

Even with Accessibility access allowed, however, I still seem to be getting the abort. Thanks for the clue, though. I’ll try to continue investigating.

[teddywing]

Using a more stock macOS 10.12 system, I was able to successfully build and run the program. I think it has something to do with my configuration rather than the OS version.

For now, I’ve determined that the TISCopyCurrentKeyboardInputSource() call at https://github.com/rvaiya/warpd/blob/567205bcc67718b5c5e75f32564dcfceffa51f43/src/platform/macos/input.m#L208 seems to be causing problems for me, but I’m not sure if that’s because I have multiple input sources that I switch between, or because I have a custom input source in my list (https://github.com/teddywing/QWAZERTY/), or some other reason.

[rvaiya]

The calls used for extracting layout data aren't well documented (I believe I found them by studying source I found online, so they may well be internal). Your exotic input setup is indeed likely the culprit. I'm not on my macbook right now, but I'll experiment when I get some time.

[rvaiya]

Can you see if the latest patch fixes it?

[teddywing]

Thanks very much for the quick response!

Unfortunately, your latest https://github.com/rvaiya/warpd/commit/714d97747cc15370e47eb6eded879425fa5d8609 patch doesn't seem to fix my abort problem. I also tried removing my custom layout from my sources list, leaving only one stock input source, but that didn’t change the result.

I tried on a macOS 12 system, and it does work there, even with my custom input source.

I’ll see if I can gather any more information.

[teddywing]

There might be an issue with running TISCopyCurrentKeyboardLayoutInputSource() in a thread. If I move the call to osx_run in src/platform/macos/macos.m, I don’t get the abort.

I also noticed this comment in MacOSX10.15.sdk/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Headers/TextInputSources.h:

Mac OS X threading: TextInputSources API is not thread safe. If you are a UI application, you must call TextInputSources API on the main thread. If you are a non-UI application (such as a cmd-line tool or a launchagent that does not run an event loop), you must not call TextInputSources API from multiple threads concurrently.

[rvaiya]

Thanks. I probably should have been more scrupulous in checking which APIs are thread safe. The macos port was written in haste and I spent most of my time learning Objective C (which you can probably tell is not my mother tongue :P). Odd that this issue has never bitten me (I still can't reproduce it, even using your custom input sources).

Can you test the latest commit?

[teddywing]

I tested 5971cfbf and it works! I no longer get an abort and the program launches successfully. Thank you so much!

I’ll close this as fixed.

I probably should have been more scrupulous in checking which APIs are thread safe. The macos port was written in haste and I spent most of my time learning Objective C (which you can probably tell is not my mother tongue :P). Odd that this issue has never bitten me (I still can't reproduce it, even using your custom input sources).

No worries :) I had the problem on one machine & OS version but not another, so it seems tricky to reproduce.

Thanks a bunch for writing Warpd. I’ve been dreaming about a keyboard-controlled mouse like this for a long time, and am so glad to be able to use it.