Adaptive mode

[mondalaci]

Adaptive mode will enable Agent to detect the actual foreground window and change the keymap of the UHK accordingly.

First up, we'll have to query the current window of the operating system. A long time ago, I was searching for such node modules, and couldn't find any, so I started writing current-window-linux, but it's not complete, may not be very stable, and it must be wrapped into a node module.

In the spirit of modularity and code sharing, I propose creating a global npm module called current-window which will depend on the OS-specific current-window-linux, current-window-macintosh, and current-window-windows packages, all yet to be created.

The interface of these modules should be a simple getCurrentWindow() function which returns a object containing all the properties of the current window that can be extracted from the OS. The properties of the toplevel current-window module should be the union of the OS-specific modules.

Actually, I'm wondering whether there's a callback mechanism in various OS-es to detect the change window event. That'd be more efficient and faster than polling. If there's such a mechanism, it should be exposed via an API.

There'll be configuration options under a dedicated menu item for this feature named "Adaptive mode" under the Agent menu. The above will take quite some work, so I'll specify the UI when we'll get there.

We will also have to update the Adaptive mode (triangle) icon of the LED display based on whether adaptive mode is enabled.

[laxu]

This will need OS-specific implementations because the methods to get the current active window are different.

For Windows you might be able to use the GetActiveWindow or subscribe a global hook to monitor the WM_ACTIVATE event. I don't know anything really about Windows programming so there might be better and more modern ways to do it.

For OSX the simplest way seems to be to just use AppleScript to figure out what the frontmost app window is. There are probably ways to detect the window with Objective-C as well.

How should adaptive mode set which keymap to use for each window? It might be easiest to build a map of application and keymap that could be managed in Agent.

I use a mouse gesture app called StrokesPlus on Windows that has a feature that allows detecting a window by dragging a reticle over it. It gets several identifiers like executable name, window title etc that can be used to fine tune if the gestures work on the whole app or just one of its windows. Something similar might be good for adaptive mode.

[mondalaci]

Thanks for the references! They seem very useful. The simplest solution may be to look into the source of existing open source applications that offer such functionality. I know that there are a lot of such apps on Linux and I can name a few. Not sure about other OSes.

Yes, Agent will allow for the user to create applications to keymap mappings, and then the firmware will expose a switch keymap command via USB which Agent will call upon window switching.

StrokesPlus probably uses relevant APIs. Like I said, it'd be useful to extract code from relevant apps. Not sure whether this one is open source and has a compatible license.

[Hurricaaane]

If I were to do some research based on open source software, I'd look at Open Broadcaster Software or Open Broadcaster Studio, which is a popular tool used by online streamers (i.e. Twitch). If it matters they are using GPLv2.

• Their method to find the process name on Windows, and a handle to the foreground process.

  • It's interesting to see that their call to OpenProcess is obfuscated. According to comments I've found, it's to deal with antivirus software, but there may be more to it.

• If you search for find_window there's some interesting information for platform-specific implementations.

• For some features, they are going through the trouble of doing window title matching.

---

Just as an anecdote, on October 2015 one of the programs I have created was wrongly detected by Valve Anti Cheat (VAC) because it was reading the name of the foreground process. Thankfully it was confirmed as a false positive and I was unbanned in less than a week. I probably won't use Adaptive mode due to paranoia...

The implementation that I used was mostly copied from this StackOverflow answer. The OpenProcess with the PROCESS_VM_READ flag (that OBS does not use) was likely a contributing factor to the red flag.

Nowadays I'm using the following instead, which is bloody expensive but I'm not willing to take any risks:

    WinDef.HWND hwnd = User32.INSTANCE.GetForegroundWindow();
    User32.INSTANCE.GetWindowThreadProcessId(hwnd, pointer);
    int pid = pointer.getValue()
    // when pid changes, call this:
    Process pr = Runtime.getRuntime().exec("tasklist.exe /fo csv /nh /fi \"PID eq " + pid + "\"");

[mondalaci]

@Hurricaaane Thank you, this is gonna be tremendously useful when implementing the current-window-windows module.

[mmroczka]

What is the status of this item? This functionality (application-specific keymaps) is listed on the home page of the UHK website. Is this finished and I just don't see it? How can I get my keyboard to recognize a specific application (like Intelli-J in the example in the video on the landing page) and automatically switch to that keymap?

[mondalaci]

@mmroczka We haven't started to implement this feature yet, and given its complexity it will take quite some time. We're doing our best, but we have to deal with a lot of other, higher priority issues. We'll definitely make it happen, but not ETA yet.

[mmroczka]

Thanks for working diligently towards making this an awesome product! Do you have any sort of public roadmap so we can get visibility into higher priority issues that are planned to be fixed?

[mondalaci]

Thanks for your understanding! We don't have an official roadmap, but recently, we've been focusing on implementing various recovery modes. In the near future, add-on specific features will take priority, and finally, extra features like this one.

[cboebel]

Just checking in since it's been awhile. Any progress on this?

[mondalaci]

Nope. If we'll make progress on this, we'll update the issue.

[mondalaci]

@cboebel Please thumb up the first comment of an issue if want to sign your interest. Repeated queries tend to me slow down things and increase noise.

[cboebel]

I've been giving this some thought and would like to try to hack on it a bit. Please forgive a potentially naive question, but.. if you suppose that active window information is available, how will the keyboard obtain this information? Do you envision agent notifying the UHK when the active window changes?

Thanks!

[mondalaci]

Yes, the idea is that Agent will detect active window changes, match window properties, and map the actual window to the preferred keymap. Then Agent will send the UHK a keymap change command featuring the relevant keymap.

It'd probably be worth to store the mapping information in the EEPROM of the UHK for portability purposes. It's doable, but there are some blocking issues, so we should implement this part later.

As for the next step, we could badly use the aforementioned current-window-{linux,windows,macintosh} npm packages. Ideally, these packages should be as efficient as possible by exposing a callback based API (whenever possible based on native system APIs) that doesn't utilize polling, and provide as many window related properties as possible for the most accurate matching.

[cboebel]

OK. I'm going to take a look at the current-window packages. Agreed on the callback-based API.

[VinceG3]

If anyone is interested in a workaround while waiting for the feature to be developed here's what I came up with on Linux. I statically compile switch-window.js with https://github.com/zeit/pkg so that it can be run by any user. Be aware that HID.node must be present in the same directory or the window will switch but the keymap won't. This means cloning the repo, running npm i then npm run build then retrieving HID.node from node_modules. Then I have a script with these contents:

#!/bin/bash
case $1 in
    chrome)
        /usr/bin/wmctrl -a Chrome
        /home/vince/bin/switch-keymap-linux CHR
    ;;
    sublime)
        /usr/bin/wmctrl -a Sublime Text
        /home/vince/bin/switch-keymap-linux SUB
    ;;
    term)
        /usr/bin/wmctrl -a vince@
        /home/vince/bin/switch-keymap-linux TRM
    ;;
    slack)
        /usr/bin/wmctrl -a Slack
        /home/vince/bin/switch-keymap-linux SLK
    ;;
    agent)
        /usr/bin/wmctrl -a Agent
        /home/vince/bin/switch-keymap-linux AGT
    ;;
esac

wmctrl is a linux program that allows you to switch window focus. I then bind this to Super-1-9 in Gnome Settings -> Devices -> Keyboard. Then I bind keys in every keymap's Fn layer to Super-1-9.

What's nice about this is that I can just pick up my mouse and use it normally. I have the right Mod key that always switches back to a normal QWERTY layout in every keymap.

I'm settling on the idea of using the Mod layer to control the application, and Fn to control the OS. Each application-specific keymap either has a QWERTY base layer if it's text-heavy, or a navigation-focused base layer if it's primarily mouse-commanded, like browsers.

I'm eventually going to extend this by treating individual browser windows as applications with their own keymaps. I'll keep each browser application in a particular tab position and just send a browser event to switch to it after I switch to the browser.

I think UHK could provide a statically-compiled version of the switch-keymap.js utility, not entirely sure how easy it would be to make it one executable, my solution requires two files. Static compilation is needed because when Gnome runs it it won't be as your current user and so it will be executing with system node, assuming it exists, rather than your NVM version. Statically compiling it yourself requires knowledge of how NodeJS works and is finicky because NodeJS is an immature ecosystem. For instance the codebase did not tell me which version of Node to use until it failed, necessitating a lot of troubleshooting to figure out what to blow away so I can start over with another version.

But that would be an easy way to roll forward without any heavy lifting required until the Agent can be extended to detect the currently-focused window. You're going to want to use the UHK to switch windows anyway, right? So hook on that instead of the window manager event. Way cleaner in my opinion.

On Windows you can easily replicate this with AutoHotKey, and OSX can very likely switch window focus with AppleScript.

One thing that would make this easier is to make it so a macro action can switch keymaps. Then I won't need the above bash script or switch-keymap at all, just a set of keymaps in Gnome to invoke wmctrl directly. But this would require development of a probably-niche feature.

[mondalaci]

@VinceG3 Thanks for sharing your solution! Even though adding keymap switch actions to macros is not planned, @kareltucek's UHK firmware fork may already support it.

[kareltucek]

Yep, it supports it.

[richrd]

I've been tinkering with making a poor mans adaptive mode for Linux. I'm using the node-x11 npm package to detect when the currently focused window changes and that seems to be working well now. The next part is obviously making it switch keymap depending on the active window.

I'm trying to import part of the agent code into my script/project so that I can call the appropriate function to change keymap (similarly as in switch-keymap.ts). For example:

const { operations } = Uhk(argv);
await operations.switchKeymap(keymap_name);

I'm running into some issues though. I've tried setting up a project with a package.json and adding agent as a dependency like this: "uhk-agent": "https://github.com/UltimateHackingKeyboard/agent.git". That pulls the agent repo into node_modules but I'm unable to successfully import any TypeScript files from there. Specifically I'm trying to: import Uhk from './node_modules/uhk-agent/packages/usb/src'; but that fails with these errors:

node_modules/uhk-agent/packages/usb/src/parse-logging-options.ts:1:28 - error TS2307: Cannot find module 'uhk-common' or its corresponding type declarations.

1 import { LogOptions } from 'uhk-common';
                             ~~~~~~~~~~~~

node_modules/uhk-agent/packages/usb/src/uhk.ts:2:28 - error TS2307: Cannot find module 'uhk-common' or its corresponding type declarations.

2 import { LogService } from 'uhk-common';
                             ~~~~~~~~~~~~

node_modules/uhk-agent/packages/usb/src/uhk.ts:3:56 - error TS2307: Cannot find module 'uhk-usb' or its corresponding type declarations.

3 import { UhkBlhost, UhkHidDevice, UhkOperations } from 'uhk-usb';
                                                         ~~~~~~~~~

It doesn't seem to know where to look for the additonal packages. I understand this might be getting off topic, but If anyone has any suggestions on how I could get this working nicely it would be greatly appreciated! If all else fails I might just have to fork agent and add my code to the forked repo, and get everything compiled properly that way.

[ert78gb]

it would not work, because agent uses typescript, we use lerna to link the packages together, so the npm i will break

Try the following steps

• checkout agent as a normal repo
• run the npm i, npm run build commands
• if the agent starts with the npm run electron command then we are fine
• you need the uhk-common and uhk-usb packages, link these packages to the global node packages cd packages/<package-name>; npm link.
• link the packages to your project cd <your package>; npm link <package-name>

n.b.

• If you run the npm i in your package linked packages will remove, you have to rerun the last step. It is the quickest workaround that I know

[richrd]

Thanks for the help! I tried creating my own separate package that depends on the packages within agent. That ended up feeling like a bit too much work to maintain since It's mainly for myself (but I also wanted it to be shareable). I ended up just forking the agent repository and adding my package within it. That way I can directly take advantage of the agent build process and the packages it provides. It's also trivial to rebase since I didn't have to modify any existing files.

Anyway I now have a fork of agent which has a new package uhk-adaptive-mode. It can be used as a simple CLI tool for adaptive mode on Linux. It requires the entire project to be built first and also a JSON config file for which programs should use which keymaps. I hope that some of the code can at least be helpful when adaptive mode is officially implemented for Linux.

I can write some basic instructions if anyone wants to try it out. I recall at least @krokofant asked me to share my POC :)

[Alexmakes]

What is the current state of this feature as a V2 customer I am interested in knowing what happening with regards to the RGB and whether or not this will be a part of adaptive mode for Windows. Ie a Photoshop layout and highlighting used in Photoshop much like this. The visual cues of the keyboard indicating it's use for the software in use would be greatly appreciated accessibility feature. If the LCD manages to get custom text written on it could be a possible to indicate through the LCD which software is is adaptive mode attached to as a thought.

[mondalaci]

@Alexmakes Adaptive mode is not a priority right now. For the time being, you can manually switch keymaps, like a Photoshop keymap. Currently, only functional backlighting is implemented, but if there's sufficient demand, we'll implement individually customizable per-key backlighting.

[Dagon]

A much simpler version of this would be helpful to me - just switching keymap when connecting or re-connecting to a different OS. I have 2 UHKs on docking stations where I use either a macbook or a Windows laptop (and sometimes play with Linux as a core OS, but that's rare - it's easier to just remote into those servers). I'd LOVE it if I didn't have to manually switch keymaps for MacOS (the non-default).

[mondalaci]

@Dagon Your suggestion is planned.

https://github.com/UltimateHackingKeyboard/firmware/issues/531 is also relevant.

[ashtonian]

Switching keymap when connecting to a different host via a kvm would be awesome. A special cross os modifier key map target would also be nice. IE ctrl when connected to windows and cmd when connected to macos.

[andriysavin]

I'm wondering if this feature could automatically switch the layout when switching between host MacOS and guest VMs like Windows in Parallels VM.

[mondalaci]

It wouldn't because Agent runs on the host OS, so it'll only be able to query the active window in the host OS, not the guest OS.

[andriysavin]

@mondalaci maybe, VM window (space?) name could be used for that?

[mondalaci]

If the virtual machine window's name reflects the current application in the window, it's possible.

[xqin]

i made a npm package to support set uhk keymap from cli and auto set keymap by current os, if somebody has the same scenario, take a look.

https://github.com/xqin/auto-switch-uhk-keymap

---

Background: I use a NUC 12 and a MacBookPro with a DELL U2723QX monitor at the same time. This monitor supports KVM function, so when the monitor signal is switched to the corresponding machine, the keyboard will be automatically connected to the corresponding machine, but my Windows and macOS Different keys are configured, so when switching systems, I hope the keyboard can also automatically switch to the corresponding keymap, so based on the code in the this repo, I wrote this small program for automatic switching.