API for probing UHK state

[soraxas]

Hi! Are there any api for probing the UHK states (e.g. querying the current pressed keys, keymap, layers, etc.) from the OS (e.g. from a bash script)? A related question is whether there are ways for macro to trigger scripts on the OS (e.g. running a python scripts)?

I know it is hard for it to have a unified interface across OS, so I'm thinking the latter might be something like setting up a path in agent, and some key press will send a signal to agent to tell it to execute a script on the OS

[soraxas]

Initially I was thinking if something like that exists, I can create some on screen display that updates the current layer in real-time.

But I see that OSD is already being tracked in #1227, so this is a general question wrt querying UHK states

[mondalaci]

Hi there!

Querying the current keys would be a security risk, allowing for keylogging.

It's not possible via the USB protocol to directly invoke OS commands from devices such as the UHK, only indirectly, via keyboard events, such as Alt+F2, "cmd", Enter, which are already possible via regular UHK macros. If you want to use an API, you can invoke OS commands from your script.

We have some related scripts in agent/packages/usb which are executable after building Agent, mostly get-device-state.ts

$ ./get-device-state.ts 
isEepromBusy: no | areHalvesMerged: no | leftKeyboardHalfSlot:LeftKeyboardHalf | leftModuleSlot:KeyClusterLeft | rightModuleSlot:TrackpointRight | layer:base 

We'll expose a well-documented API eventually. It'd help shape the API if you elaborated on your exact use case.

[soraxas]

Hi Mondalaci thanks heaps for your swift response and detailed explanation

Yes, I completely understand the security implication of being able to query key states. In that case, how would OSD be realised within agent (would the key states only be available internal to agent, via some secret handshake)?

AFAIK I don't think macro can trigger shell scripts directly in agent, isn't it? You can only indirectly capture obscure key-combo emitted by uhk in the OS level (as key shortcut) and then run some command.

With regards to API, I am thinking if it's possible to:

1. Query OS info from UHK's side to further enhance the macro capability. For example:
   1. I'd like to be able to detect the current OS to modify the behaviour of macro as I frequently switch system
   2. Perhaps probing current OS state (e.g. current CPU load or current active application (for fake implementation of adaptive mode)) would be nice too.
2. API access to UHK from OS side. For example,
   1. accessing key-states to implement live OSD
   2. sending command to uhk to change keymaps
   3. setting LED text with info like CPU load / arbitrary text sent from OS

[mondalaci]

Layer states are available to Agent for the purpose of the OSD. We'll also expose the current keymap to Agent, and it's also aware of the mappings. The listed information is sufficient to implement OSD.

The UHK is a USB keyboard, and as I said, USB doesn't provide a native way to invoke OS commands. The UHK could send special USB messages to Agent, which, in turn, could invoke commands, but that would require Agent to always run in the background. We don't plan to implement such a feature.

1.1. We plan to implement an installation-specific default keymap in Agent. That would enable your use case. 1.2. #660 and https://github.com/UltimateHackingKeyboard/firmware/issues/322 will enable your use case.

2. Changing keymaps via switch-keymaps.ts is already possible.

[soraxas]

Thanks @mondalaci for your detailed reply.

Thanks for the several links and pointing out the current capability. However, there might be some misunderstanding on some of my raised points:

• 1. (i): I meant it as like some sort of condition in the macro engine. Something like this: (under the macro syntax) ifOsEq linux {myCommand} .... I'm not sure how feasible this is as this post seems to suggest that there is no native way to detect OS. My usecase: Currently I'm trying to implement a vim-like normal model keymap via macro. Most common vim motions like dd, dw, ciw, yy, etc. works great. However, there is a difficulty with w, e and the likes under different OS. Inside a textbox, ctrl+right behaves differently across OS:
     • To implement w in windows you only need ctrl+right key combo
     • To implement w in linux you need an extra right arrow key, i.e., tapKeySeq C-right right macro
     • To implement e in windows you need to compensate the overshoot via tapKeySeq C-right left key combo
     • To implement e in linux you only need Ctrl-right To make it clear, I'm not trying to have installation specific default (as they also won't auto-detect OS right?), but I want to have OS-specific key/macro, exposed via macro.
• A related thought is that if the previous point is not feasible with pure USB IO, would it be possible to expose the JS engine to macro? Such an approach will relies upon the agent software running in the background, but it will expose a greater set of functionality to macro programming for a smarter keyboard. It should then be trivial to expose more info for the keyboard macro for context-dependent functionality

[mondalaci]

OS fingerprinting via USB cannot be implemented reliably, unfortunately.

Please open an issue in Karel's firmware regarding the mentioned macro commands until they're merged into the stock firmware.

We want to make the smart macros as self-contained as possible and don't plan to expose the JS engine to macros.

I'm closing the issue, but feel free to follow up regardless.

[soraxas]

Thank you for your response.

I know about Karel's firmware and I know it's not feasible to perform the OS detection purely from the macro side; that's why I've posted it here.

If it's not feasible to do OS fingerprinting purely from USB handshake, can agent pass along the OS identification info back to the firmware side? Surely this would be highly related to the advertised "adaptive" mode, as being application context-aware would imply knowing the current OS.

[mondalaci]

As for the mentioned features,

1. Agent's adaptive mode will make the UHK switch to the keymap of the foreground application. The UHK will only receive switch keymap USB commands from Agent, not OS information.
2. The Installation-specific default keymap feature will also utilize the switch keymap USB command, and Agent won't pass OS information to the UHK.

Passing OS information to the firmware is feasible, but it seems to be a very niche use case, and its use may be cumbersome; If Agent is not running, the OS firmware variable would be undefined.