Kanata sometimes crash with only "Illegal instruction" being printed in the terminal

[robinmoussu]

Requirements

• [X] I've searched issues to see if this has not been reported before.

Describe the bug

Sometimes (usually after sleep, kanata does crash and all I get is "Illegal instruction" to help me debug the issue.

Relevant kanata config

macOs Catalina (10.15.7) I’m using the kext-support of driverkit as described in https://github.com/jtroo/kanata/issues/676 The version of kanata I’m using is (apart from the driverkit dependency) the current main.

To Reproduce

Run kanata and do stuff until it crashes… I know it’s not useful.

Expected behavior

Ideally not crash, but at least display what the issue is.

Kanata version

main (1.5.0)

Debug logs

No response

Operating system

Mac OS catalina (10.15.7)

Additional context

No response

[rszyma]

It's a generic error produced by a faulty CPU instruction: https://stackoverflow.com/a/47901355

[rszyma]

The best thing you could do is to inspect core dump with a debugger and see the stack trace to see the location in code. However I've no idea how to do that, I haven't used debugger for inspecting core dumps, so I can't help.

[robinmoussu]

Silly me, I forgot to check if core dump where generated / run kanata inside gdb. I will try that.

[robinmoussu]

The stacktrace isn’t really that useful

Thread 3 received signal SIGILL, Illegal instruction.
[Switching to Thread 0x1d07 of process 15530]
0x00007fff328e450f in ?? ()
(gdb)
(gdb) bt
#0  0x00007fff328e450f in ?? ()
#1  0x00007fff3273e116 in ?? ()
#2  0x00007000074b7f40 in ?? ()
#3  0x0000000100087bbc in release_strings<__CFString const*, __CFString const*, __CFString const*> (strings=0x0, strings=0x0, strings=0x0) at c_src/driverkit.cpp:260

and here is the function in question in driverkit:

258│ template<typename... Args>
259│ void release_strings(Args... strings) {
260|     (CFRelease(strings), ...);
261│ }

The documentation for CFRelease is here.

---

I modified driverkit to add a wrapper around CFRealease(strings) to be able to inspect the arguments being passed to CFRealease

258│ template<class T>
259├>void debug(T&& arg) {
260│     CFRelease(arg);
261│ }
262│
263│ template<typename... Args>
264│ void release_strings(Args... strings) {
265│     (debug(strings), ...);
266│ }

#0  0x00007fff328e450f in ?? ()
#1  0x00007fff3273e116 in ?? ()
#2  0x000070000fa8cf10 in ?? ()
#3  0x000000010008dadb in debug<__CFNumber const*&> (warning: (Internal error: pc 0x190a in read in CU, but not in symtab.)
warning: (Error: pc 0x190a in address map, but not in symtab.)
arg=@0x7fff3273e116: 0xe5894855c3f88948) at c_src/driverkit.cpp:259
warning: (Internal error: pc 0x190a in read in CU, but not in symtab.)
warning: (Error: pc 0x190a in address map, but not in symtab.)
warning: (Internal error: pc 0x190a in read in CU, but not in symtab.)
warning: (Error: pc 0x190a in address map, but not in symtab.)
#4  0x000000000000190b in ?? (warning: (Internal error: pc 0x190a in read in CU, but not in symtab.)
warning: (Error: pc 0x190a in address map, but not in symtab.)
)
#5  0x000070000fa8cf30 in ?? ()
#6  0x000070000fa8cf40 in ?? ()
#7  0x0000000100087b46 in release_strings<__CFString const*, __CFString const*, __CFString const*> (strings=0x0, strings=0x0, strings=0x0) at c_src/driverkit.cpp:265

(gdb) p arg
$1 = (const __CFNumber *&) @0x7fff3273e116: 0xe5894855c3f88948
(gdb) p *arg
$2 = <incomplete type>

But the best I found was the swift documentation of CFNumber which unfortunately didn’t help me to find the value behind this CFNuber.

[rszyma]

https://github.com/Psych3r/driverkit/blob/e556f262ab24f1d3d93ce02267f17a51b2aa43d0/c_src/driverkit.cpp#L230

this line looks suspicious - the function is named release_strings but the args passed here are number type.

[jtroo]

Both values passed to release_strings are created by CFNumberCreate and are of the same type so that looks ok to me.

Internal error: pc 0x190a in read in CU, but not in symtab

Briefly looking online, this appears to be an internal GDB error, and may have more useful output be printed if a newer version of GDB is used.

[rszyma]

@robinmoussu how did you compile kanata? It seems like debug symbols are missing, otherwise you'd probably see some value instead of <incomplete type>

[robinmoussu]

As far as I can tell (MacOS is a nightmare to manage gcc/clang version), I’m using

• GNU gdb (GDB) 13.1
• Apple clang version 12.0.0 (clang-1200.0.32.29)

I did build kanata with cargo build which should be the debug version.

---

But things just got worse. I had kanata running inside a gdb started with sudo, and to get the info about gdb I did press ctrl+c to put the kanata process in hold (a bit like ctrl+z if I was just started in a regular console). Something went horribly wrong because ctrl+c was send ad infinitum. I did a hard shutdown (press 5+ second on the power button), then restarted my mac. And now neither kanata, nor karabiner wants to start. I did follow the procedure described by kanabiner to re-install the driver, without success.

[robinmoussu]

Whooo, I managed to properly re-install driver-kit. In case anyone find this thread, here is what I’ve done:

• Check the status of driver kit: systemextensionsctl list
• Uninstall driverkit: bash '/Library/Application Support/org.pqrs/Karabiner-DriverKit-VirtualHIDDevice/scripts/uninstall/deactivate_driver.sh'
• Reboot
• If for some reason there was driverkit listed 2 times at step 1, uninstall the second one like before and reboot again
• Normally you should get a pop-up that say that driverkit was blocked and you can allow it in system settings

Based on:

• https://github.com/pqrs-org/Karabiner-Elements/issues/2616
• https://github.com/pqrs-org/Karabiner-Elements/issues/2876

[rszyma]

Well, you just learned the hard way that you can't really pause kanata, otherwise keyboards get stuck.

---

If the current goal is to inspect specific variables, the first thing I would try is print debugging. Have you tried it already?

[robinmoussu]

The issue is that it crashes and I have no info. When it crashes the regular layout is restored as-if I never started kanata. So far I don't know what driverkit does wrong and what I should do to add more debug informations when it crashes.

[rszyma]

About the missing debug symbols: looking at the cc crate code (see https://github.com/rust-lang/cc-rs/blob/802399f3fab0820707dc8e562f206594e9a9e1db/src/lib.rs#L3252), cc::Build::new by default won't enable debugging symbols. Can you compile kanata with DEBUG=1 cargo build, cause crash and see if gdb shows more info?

[robinmoussu]

Arf, I didn’t realized that after reinstalling driverkit only karabiners works, and now I have no access to kanata. Here is what is printed in the console (both when build with DEBUG=1 or without:

robinm@MacBook-Pro:~/perso/kanata> sudo target/debug/kanata
2023-12-29T02:48:54.18952+01:00 [INFO] kanata v1.5.0-prerelease-3 starting
2023-12-29T02:48:54.189571+01:00 [INFO] You may forcefully exit kanata by pressing lctl+spc+esc at any time. These keys refer to defsrc input, meaning BEFORE kanata remaps keys.
2023-12-29T02:48:54.190635+01:00 [INFO] process unmapped keys: false
2023-12-29T02:48:54.190902+01:00 [INFO] NOTE: kanata was compiled to never allow cmd
2023-12-29T02:48:54.201134+01:00 [INFO] config parsed
2023-12-29T02:48:54.201249+01:00 [INFO] Sleeping for 2s. Please release all keys and don't press additional ones.
2023-12-29T02:48:56.204831+01:00 [INFO] entering the processing loop
2023-12-29T02:48:56.20504+01:00 [INFO] entering the event loop
2023-12-29T02:48:56.205124+01:00 [INFO] Init: catching only releases and sending immediately
IOServiceGetMatchingService error:
2023-12-29T02:48:56.241231+01:00 [ERROR] failed to open keyboard device(s): grab failed

Press enter to exit
2023-12-29T02:48:56.856212+01:00 [INFO] Starting kanata proper
2023-12-29T02:48:56.856294+01:00 [ERROR] channel disconnected

/// Here, I copy-pasted a new-line (since the keyboard didn’t work at all) since it was asking me to “press enter to exit”

Error: failed to open keyboard device(s): grab failed
libc++abi.dylib: terminating
Abort

In case it may be useful:

robinm@MacBook-Pro:~/perso/kanata> systemextensionsctl list
1 extension(s)
--- com.apple.system_extension.driver_extension
enabled active  teamID  bundleID (version)  name    [state]
*   *   G43BCU2T37  org.pqrs.Karabiner-DriverKit-VirtualHIDDevice (1.0.0/1.0.0) org.pqrs.Karabiner-DriverKit-VirtualHIDDevice   [activated enabled]

[rszyma]

In case it may be useful:

robinm@MacBook-Pro:~/perso/kanata> systemextensionsctl list
1 extension(s)
--- com.apple.system_extension.driver_extension
enabled   active  teamID  bundleID (version)  name    [state]
* *   G43BCU2T37  org.pqrs.Karabiner-DriverKit-VirtualHIDDevice (1.0.0/1.0.0) org.pqrs.Karabiner-DriverKit-VirtualHIDDevice   [activated enabled]

Isn't Karabiner-DriverKit-VirtualHIDDevice dext? And you need kext?

[robinmoussu]

Good catch. Indeed I need kext. I cannot remember what I did to install it before (when it was working)

[robinmoussu]

I tried to install the latest version which should include kext support without success (I got the same error). Is there anything special other than cargo install kanata to enable it?

[rszyma]

cargo install kanata will install v1.5.0, and the commit that added kext support came after v1.5.0. You need to install from source for the time being.

[robinmoussu]

Same thing when building from the latest main.

[robinmoussu]

@Psych3r Do you have any idea why kext doesn't seems to be used/installed?

[psych3r]

@robinmoussu, Kext should be used automatically on macOS versions 10. However, installing it has to be done manually by the user. So first, Install kext, maybe refer to kmonad's installation guide if needed on how to install it. Then confirm that driverkit is using kext. Build driverkit with cargo build -vv to see a verbose output. If kext is used, you should see "macOS version {major}.{minor}.{patch}, using kext..." somewhere in the output. And also, add .flag("-g") to build with debugging symbols. Then the backtrace should be more meaningful. Sorry I can't help more, If I had access to macOS 10 I would've debugged it myself.

[psych3r]

And no need to worry about release_strings releasing a CFNumber. It internally calls CFRelease which releases CF numbers and string references. It's just a poor choice of a function name on my part.

[robinmoussu]

So first, Install kext, maybe refer to kmonad's installation guide if needed on how to install it.

That was it, I had to reinstall kext (as described in kmonad documentation)

Kanata is working again, I can go back to debugging my original issue.