Brainstorming for usb endpoint for host commands

[skullydazed]

Recently I've seen a few use cases that would benefit from a way to send commands to the keyboard from the computer. Some examples:

• Turning an LED on or off to indicate the state of something
• Changing the RGB underlight mode as part of an alert
• Changing the layer when vim enters insert mode

This could also be the foundation for other new features if we decide to implement them in the future:

• Macro changes without flashing
• Keymap changes without flashing

I'm opening this issue to discus the merits of this idea, collect new use cases I haven't thought of, and to discus how this might be implemented. If we end up moving forward with this idea we can open a new issue to track implementation.

[fredizzimo]

I have also had similar thoughts and mentioned it earlier.

My ideas have all been more in the direction of the second set of use cases, although it supports the first set as well. I have always wanted to run the full QMK keyboard software on the host computer and let the keyboard be just a dummy device that sends the physical key presses to it.

This is not very different from what I'm already doing with the Infinity Ergodox. The second half is just a dummy device that communicates the keypresses through the serial link protocol to the master. The master then does all the complicated logic to keep things in sync.

Some information flows the other direction as well, like the active layer, and the full state of the keyboard, this is used for the visualizer, which can display things on the LCD or on the LEDs based on this information.

So if you look at it closer, it's the completely same thing. Only change the host PC to master, and let the keyboard(s) be the slave.

In order to achieve this, we first need to get the QMK firmware to compile natively. But this is needed for the full set of unit tests as well. So I suggest that we start by enabling it through unit testing. Then when it works, we could just plug in the serial_link protocol, and implement the master/slave separation, which doesn't need many changes to the actual keyboard code at all.

The serial link protocol needs to be ported to AVR and we also need to a an USB serial port as an endpoint, but that shouldn't cause any problems. However, we might want to replace the lowest byte stuffing layer and maybe the validation (CRC32) with the native USB packetizing and error correction(CRC16).

But the routing and transport protocol could be used, since it makes things easier at least when dealing with the Infinity Ergodox. They would automatically take care of routing data from an Ergodox Infinity slave, through UART to the master, and from there through the USB to the host PC, and the other way. It also allows you to target specific devices in a list of daisy chained keyboards, which at least the Infinity Ergodox supports.

For other keyboards we still have the high level remote objects that we can use, which could simplify the programming a bit. It also makes things work the same way, no matter which physical keyboard is connected, so I see the benefit from using the same library for that as well.

It would also be easy to add a message based protocol in addition to the remote objects, if we need that.

[skullydazed]

That's a much different direction than I was thinking. It's very interesting also. However, as interesting as it is, I strongly feel that we need to operate as a plain HID keyboard by default. Requiring a host driver will preclude a lot of uses that people expect- phone/tablet use, game consoles, BIOS access, etc. If we go in this direction we will need to initialize as a plain HID keyboard and have something that switches us to serial mode when the proper host driver is loaded.

There's also the question of multiple keyboards. Right now I can have different keyboards with different keymaps. I've seen this in the wild where people use two plancks for a split typing experience. I do this myself where my clueboard with split backspace gets a different map than my other clueboards. If we move the key handling logic to the host we'll still need a way to uniquely identify keyboards beyond usb vendor/device ID's.

So the question becomes, do we tie communicating to the keyboard from the host to the serial link mode? My thought right now is no.

Requiring a custom driver for input will make our users' lives more difficult. No matter how well we code the driver some users will have difficulty installing it. It also implies they have root/admin access to their computer, something we can't guarantee. The expected benefit needs to outweigh these problems. For someone who wants a lot of functionality and ultimate flexibility for their keyboard, the serial link and doing all key translation logic on the host is a clear win. They eliminate code size limits, increase their available processing power by several orders of magnitude, and reduce the need for more powerful MCU's. But what is the benefit to the majority of users who will use the default keymap or who will only tweak the keymap a little bit?

There is also the question of sending commands back to the keyboard as an ordinary user. If we operate as a HID device with no special drivers required I can take my keyboard to a computer I've never used before and install the software I need to control LED's. No root/admin access required. For many users that may be as simple as plugging in their flash drive and double clicking their portable version of $SoftwareApp.

If we allow QMK flashed keyboards to operate in slave mode, and I think we should, I believe it's a separate question from opening a second HID endpoint for keyboard control.

[fredizzimo]

I agree that the keyboard should still work as a standalone keyboard without any extra drivers or software running on the PC.

The system I'm talking about would connect to the PC software only when available, otherwise it would run the normal flashed firmware.

One of my intended use cases would be to use it as a development tool. Since the whole QMK would run on the PC, it means that you can debug the code using a normal debugger. You can also avoid all the hassle of uploading the new firmware each time you make modifications, instead you can just restart the program. And you can also do as many modifications as you like, without risking getting ruining your chip by flashing it too many times.

Once you are happy with the keymap, you would just flash it to the keyboard, and the host software would not be needed anymore. That of course if you keymap uses only features that are available in QMK itself.

But of course if you are going to send back stuff to the keyboard, like the LED control that you are mentioning, then you need something that runs on the PC anyway.

In any case, I don't think we need to write any driver, we could use Raw HID, which the debug output through HID listen already does. Or alternatively use a virtual serial port. So the requirements would be a single executable, that doesn't need anything extra.

Another thing that this host software could do is to provide some sort of socket based API, so that we could control it further from other software. You gave VIM as an example, so using https://github.com/Shougo/vimproc.vim, you could send socket commands which are then interpreted by the software and sent further on to the keyboard if needed.

Regarding support for multiple devices, I believe we could use the USB descriptors, in the case of raw HID. Otherwise, moving the device id from the config.h file to the makefile, so that it can be controlled by a parameter when building, sounds like a plausible option.

I'm not sure exactly what your alternative would be. But even if we used say the LED page (0x8) and the Alphanumeric display page(0x14) for LED and LCD control respectively, we still need something on the host to control it, and we would have a lot less control, so I don't see that option as attractive as my proposal. That said, there's still a lot that we could do with that as well.

[algernon]

Wasn't there a Virtual Serial Port thing that got merged recently? If that could be used to communicate with the keyboard, that would be one way to do it. (I tried doing that and failed, but I only spent like 10 minutes on it so far)

Then, it would be up to the keymap to interpret the commands that come in - QMK could provide some helpers for common things, but ultimately, the keymaps would be able to define their own protocol over the virtual serial port.

My use cases include:

• Switching layers depending on the mode in my Spacemacs.
• Switching layers depending on the active window.
• LED controls.
• Turning debugging on programmatically.
• Semi-dynamic macros: instead of recording them on the keyboard, I'd "upload" the macro via VSP. This may need a small interpreter in the keymap, but I'm fine with that.

[skullydazed]

@fredizzimo Glad we're on the same page re: HID keyboard. :)

One thing that isn't clear to me is how keys would be typed in slave mode if you didn't have a driver. Would the host program have to send key presses on a roundtrip back through the keyboard so they could be typed over USB? This is where the driver comes into play in my mind.

I'm not sure exactly what your alternative would be. But even if we used say the LED page (0x8) and the Alphanumeric display page(0x14) for LED and LCD control respectively, we still need something on the host to control it, and we would have a lot less control, so I don't see that option as attractive as my proposal.

My original vision (which was admittedly light on details) looked something like this:

• Keyboard/keymap author registers one or more functions as commands on the control bus, for example they might register the cmd_led_set(int led, int duty_cycle) function as the command "led_set".
• These functions would be responsible for "doing something" (EG, set pin D2 high, change the duty cycle on TIMER1 to 50%, etc)
• The host program would send commands (led_set 1 512)
• QMK would read the command and dispatch to the appropriate function (EG, cmd_led_set(1, 512))

That would provide a loose framework for communication from the host to the keyboard, while not tying functionality to any particular keyboard design or paradigm.

That being said, if we can do this entirely in userspace using slave mode, I'm on board with that. If it works like I suspect above, where keypresses are sent back through the MCU, I'm concerned that lag might be an issue but I don't have a concrete foundation for that concern.

[fredizzimo]

@skullydazed, I left out too many details too.

Regarding sending the actual keycodes to the OS. I have two options

1. We could do like autohotkey does for example, and generate keystrokes through the operating system APIs. That is operating system dependent, and might not work in all situations, for example when some application reads raw input.

2. Like you suggest, roundtrip the input back to the keyboard to have it generate the real USB input codes. Unlike you I'm not the slightest worried about lag though.

   First the USB poll rate is 1000 times per second or once every ms, so that means that we have a one way delay of 0-1ms, we can ignore the wire delay since it should be just nanoseconds (http://www.anandtech.com/show/2803). This means that if we send the data to the PC, back to the keyboard, and finally to the PC again we get a delay of just 3 ms.

   To put that into some perspective Cherry MX specifies a maximum debounce time of 5ms. So by default we already have a bigger lag than that, and that doesn't seem to hurt us.

   But there's more, the best monitors have a lag of 10ms. While TVs could have over 100ms. I haven't heard anyone complaining that using a keyboard on a TV screen doesn't work, and surely adding 3 ms to those lags shouldn't matter.

   But what about games. It turns out that this generation of console games have really bad lag at minimum around 80ms. And the monitor lag is added on top of that. There are some hardcore reaction based fighting games on the list, which tells me that even this type of lag is not too bad. So I think we can safely say that the 3 ms extra lag won't hurt us at all.

   A final comparison that we can do is to take someone who types 200WPM, or 1000 characters per minute or one character every 17ms. The added delay is still not even close to that.

   Even if we use a polling delay of 8ms, like the Anandtech article shows as a maximum, we still end up with only 24ms delay, which I very much doubt is a problem.

   The only potential problem could be if we connect multiple keyboards to the same PC, and they have different latencies. That could cause mistakes to happen if keys gets pressed in the wrong order. However as long as both keyboards takes the same route, the difference should be very small, at least smaller than the default debounce time, so I doubt that would matter either.

A also left out the part that describes the commands from the PC to the keyboard. Your way is one way of doing it. Using the visualizer, like I do for the Infinity Ergodox is another option. Actually I will soon write a proposal for doing a much more generic visualizer than what I have currently. So more about that, when I get it written down and open the issue.

@algernon, yes I don't know how I forgot that already... #677 adds a virtual serial port which plover could use. Personally I would like to see something more generic than having each keymap doing things by their own though.

But those changes could definitely serve as a foundation for this.

[skullydazed]

I'm not a fan of OS specific code to generate keystrokes, it means we need to perpetually have maintainers for each OS. Since the latency isn't an issue (and even if your estimate was too low by an order of magnitude I still think that is acceptable lag) my opinion is that sending keystrokes back through the keyboard to be sent over the USB channel is better.

I am worried about what happens if the host program crashes. It would be good to work out some sort of watchdog functionality to return to HID mode in that situation.

Re: commands, I think it would be good to define some default commands/API's (Backlight behavior, RGB behavior, LCD behavior, Speaker behavior) but give keyboards flexibility in how those are implemented and the ability to define new functionality we haven't thought of.

[fredizzimo]

@skullydazed, I'm glad you are not a fan of that idea, as I'm not either :smile:

I think it would be fairly easy to detect that the host program crashes, as the USB connection should go down. The handling could be very similar to what we do an a suspend, just reset everything. The keyboard should then be able to continue in normal HID mode.

We probably need some sort of macro to force the keyboard into HID mode, even if the host is there as well. And maybe the other way around too, to make it completely non-functional without the host.

And yes, I agree that keymaps should be able to completely customize things if needed.

[skullydazed]

The macro is a good idea. At first I wasn't sure why you'd want to force slave mode, but then I considered trying to debug the host program when the keyboard keeps reverting to master, and it made more sense. :)

I'm afraid I have a lot to learn about USB still, so I don't know what the device sees when a program closes the serial port. If it will notice when the host side closes the serial port that's good enough for me.

[skullydazed]

It seems like we've reached something of a consensus. Does this look like the rough outline of how to do this?

• Get the test infrastructure going
• Add support for operating in slave mode communicating over the serial port from #677
• Add support for running qmk on the host communicating over serial
• Create some standard interfaces for things like backlight, LED control, audio, and other common hardware
• Develop an API that keyboards and keymap authors can use to implement their own commands

[fredizzimo]

Yes, the list looks correct to me.

[IBNobody]

My 2 cents...

• Don't use the USB CDC class.

If the virtual serial port appears as a COM port in windows (which I suspect that it does), I recommend avoiding using it as a general system. Windows will remember ghost serial ports and reserve a COM port for said serial. While it is easy to wipe out ghost serial port devices, it is not as easy to remove ghost serial port COM assignments. If I have 5 QMK keyboards that I randomly hook up, that is 5 COM ports that I have to deal with. Plus, Windows will eventually run out of COM ports.

• Allow mixed mode communication.

We should set the framework up to be able to work alongside the existing keyboard or NKRO channels rather than an all-or-nothing slave mode.

For my current AHK implementation and goals, I want to have my keystrokes sent via the standard or NKRO devices and have a secondary channel to send every keystroke or a subset of keystrokes via the serial channel. I would also want to receive status control data via the serial channel.

[refactorized]

Two Thoughts:

A) The Keyboard should be an input device first and always, and it's ability to process messages from the host should not be in any way coupled to this functionality. I think there is consensus on that already though.

B) A simple serial connection, com port, whatever would be great for the fact that any program on the host can throw data at it, driver free. If that is not an option, then at least having a very thin, very simple service on the host as say a named pipe, would be great too. The point being, it should be easy to install driver software 0-1 times then proceed to have any program utilize the functionality with a very simple protocol. I should be able to easily write an atom editor plugin to change lights based on the error status of my code knowing only javascript and nothing about QMK outside of a well documented API. That would be one hell of a feature.

[jackhumbert]

I was been meaning to comment on this as well - thanks @refactorized for pinging this :)

I've been messing around with using Midi SysEx to communicate with the host - I've written a little tray app that listens/connects to the keyboard after a handshake. It works pretty well, and I've had success with transmitting Unicode as well. Midi's supported by just about every device/platform, so porting something similar to everything should be pretty easy (I'm only working in Windows right now).

I'll have some more info/a dedicated PR/issue for it soon, but if you're interested in seeing what I'm working with, checkout the wu5y7 branch - my focus is on the Ergodox EZ right now (with multicolor LEDs, etc), so others may not compile.

[refactorized]

@jackhumbert That sounds pretty cool - are you using MIDI in both directions or just from the host back to the keyboard? Is there a reason you are using SysEx which is basically packets over MIDI, which is a protocol sitting on top of a serial protocol, versus just plain old serial/tty back to the KB? It seems that SysEx would be a specific application of a more general capability - in any case that's great news.

Unfortunately, I can't test any of this out, I do not possess any QMK devices (yet), but have been looking into something along the line of the ergodox or K-Type ( eventually compatible I assume ).

Thanks for the quick reply!

[jackhumbert]

Yeap! It sends both ways - there's actually a pretty decent addressing protocol built in because of the THRU port functionality (basically daisy-chaining devices), which obviously isn't needed here, and is ignored.

[refactorized]

I hope this ends in keyboards with sliders and x/y controllers

[algernon]

A keyboard with sliders, volume knob, and perhaps a joystick-like thing instead of arrows, in an ortholinear fashion. I'd buy that.

[wilba]

It's funny that I just forked QMK to implement exactly this kind of functionality (keymap editor app), and the first notification I get is this thread ;-)

I'm still undecided which way to proceed, virtual serial or raw HID.

Virtual serial is a lot more portable. Apps can be writted in practically anything... Java, Python, even AutoHotKey. If people want to hack a bit of script to make their keyboard backlight flash when they get an email, then this makes it a little bit more accessible.

Raw HID is a bit more "polished" a solution. The required functions to send/receive data could be wrapped so it could be used in code other than native-compiled C/C++

Maybe it's worth doing it both ways, and let the end user (in this case, the person hacking QMK to do cool stuff) chose which way they want to send/receive the data, but keep the protocol the same. Ideally, that protocol should support core QMK I/O as well as keyboard specific I/O... i.e. if a given keyboard implementation wants to have special commands for setting backlight effects, the QMK core passes on those commands to the keyboard implementation.

[refactorized]

So I guess my big question is can the keyboard enumerate as both an HID keyboard and a serial port, midi port, etc. at the same time, without something like a hub?

[IBNobody]

So I guess my big question is can the keyboard enumerate as both an HID keyboard and a serial port, midi port, etc. at the same time, without something like a hub?

It can be done now. The problem is that the system is limited to 4 endpoints. Right now, there are endpoints for the normal keyboard, the mouse emulation, the "system" keys (mute, etc), the raw HID console, NKRO, MIDI, and virtual serial. You can only choose 4 of these 7.

We could merge NKRO with the normal keyboard since it is an either/or setup. (NKRO may even be able to handle the system endpoint, but I would have to look at the HID spec again.) That would leave us with fewer endpoints to have to choose from.

@Wilba6582 If I had to choose one... My vote would be for Raw HID, and I already have a DLL wrapper that works for one-way communication. This is how my keyboard helper script works.

However, I agree that the best solution would be to abstract the Vert Ser / Raw HID / MIDI stuff out into a single interface. Throw in I2C and raw serial, too. The user would select the interface they wanted and compile their code to use it.

[IBNobody]

Also, the rawhid library is available on all three major OS'es.

[wilba]

@IBNobody I didn't know about the 4 endpoint limitation. I'd prefer not to reuse the "Teensy" raw HID endpoint, though. Leave that as is (for debugging/HID Listen), add a new one with new (user-configurable) vid/pid and a defined protocol. Ideally, it would have a "QMK" vid/pid so host apps only need to know about the one vid/pid, and the "QMK" protocol would allow querying what specific keyboard/firmware it is. Thus a host app could iterate over connected "QMK" devices and control a specific one. (FWIW debug output could be redirected through this "QMK" raw HID, a custom "HID Listen" made, and the "Teensy" raw HID could be retired.)

edit: Ignore what I said about a "Teensy" VID/PID... I meant the Usage Page/Usage magic numbers in the console descriptor.

[IBNobody]

@Wilba6582 Regardless, the first thing I would need to do is get computer-to-keyboard communication working. That isn't operational at the moment. Then I could abstract out the endpoint and allow it to be customized.

[wilba]

@IBNobody well, that's also the first thing I need to get going, too :-)

[IBNobody]

@Wilba6582 Going to tackle it this weekend using my Atmel debugger.

[wilba]

@IBNobody HIDAPI looks better than PJRC's Raw HID: https://github.com/signal11/hidapi especially its functions for enumerating devices/interfaces. After a bit of massaging, I got it to receive "console" output from QMK. (The trick is finding the one device which has the "console" usage page/usage).

[PureSpider]

I successfully used the java bindings for hidapi over at https://github.com/gary-rowe/hid4java to read my keyboard's console output in java - I can provide code for that if needed.

[wilba]

I got the raw HID working in QMK, reusing the "console" code, I got QMK to receive data and do things like change the backlight. When I refactor it out into a new interface, I'll make a branch in my fork for others to have a play with it.

[jackhumbert]

Nice - I'll abstract out the receive/send methods of the protocol for other things to use.

[wpc009]

I think something like the serial console in kiibohd is a good solution.

[iFreilicht]

What's the current status on this?

It can be done now. The problem is that the system is limited to 4 endpoints. Right now, there are endpoints for the normal keyboard, the mouse emulation, the "system" keys (mute, etc), the raw HID console, NKRO, MIDI, and virtual serial. You can only choose 4 of these 7.

Why do the system keys need a separate endpoint? Those are part of the standard HID usage tables, are they not?

What limits us to 4 endpoints? (2 bits?) Any idea how much work it would be to increase that limit to 8?

@Wilba6582 HIDAPI looks pretty damn good. I believe a polished solution like this is absolutely necessary for robust support of multiple keyboards and maybe a general QMK configurator GUI. @refactorized, what sort of functionality would be required host-side to implement editor plug-ins? You say a named pipe to send API commands too would already be enough, and I guess all editors that do support plug-ins allow some sort of command-line calls, so any kind of command-line tool would be sufficient, right?

[wilba]

@iFreilicht I've been using the raw HID interface and HIDAPI to configure keymaps and backlight settings on Zeal60 with reasonably good success... I've just been a bit slack on implementing a GUI for the host side... but in theory what works for Zeal60 can work for others. The hard limit is EEPROM size for storing the keymaps :-(

[iFreilicht]

Very cool! Well the keymaps can be stored in flash as they are now, I'm not seeing a big problem with that. Sure it's a little less convenient to change them, but it's also a lot more flexible with what you can do on a single keypress. Anything that calls a custom function could not be stored in EEPROM anyway.

[fredizzimo]

I don't see storing keymaps in eeprom as a viable option at all. @iFreilicht alread commented about storing function pointers there. Even if we somehow manage to put the right adresses there, they might change the next time you compile a new version.

But we have a similar problems with, especially the QMK keycodes, they are not guaranteed to stay constant. And varies depending on which features you enable, and can also change when the QMK code is changed.

Finally keymaps can consist of so much more than just layer definitions so they are not very flexible.

In other words the keymaps can easily get corrupted, so I don't recommend that.

[wilba]

I've already implemented keymaps in EEPROM. It's perfectly viable. As long as the tool which programs keymaps is compiled with the same int values of keycodes, there is no problem, and I ensure this with version numbers stored in the code and EEPROM.

[fredizzimo]

@Wilba6582, I checked your implementation in the Zeal60 branch.

However I don't see how it deals with changes to for example quantum_keycodes.h, other than also having to bump the eeprom version when that happens. In a similar way if options are enabled or disabled, the eeprom version also has to be bumped, and the tool need to have the exactly same features enabled.

Even if extreme care is taken to bump the version when changes are made, then it's still incomptible with dynamically enabling and disabling features from the make commandline.

And when an incompatibliy version change happens, the keymap reverts back to the default, which most likely causes confufsion at least for inexpirenced users. And in this case to you need to update the EEPROM to get it the keymap back. So that's two things to do instead of one. I also hope that the tool you use for writing these EEPROM values have some kind of versioning and backup, as otherwise the keymap is lost.

I'm not telling you that EEPROM keymaps are useless, they might very well be perfect for you. But as a general replacement for normal keymaps they are not good and only complicates things. Personally I don't see any use cases, other than perhaps to help building the keymap, but once that's done it should be copied back to the flash memory. But even for building a GUI configuratior, I would rather have it generate the keymap.c, perhaps compiling it on some web server backend, than to store things directly in eeprom. That way it can also configure more advanced functionality than just keycodes.

[wilba]

Zeal60 users can either a) install a pre-built, versioned .hex and use a tool on the host to program keymaps and backlight settings, or b) compile their own firmware with whatever features they want (including the keymap in flash memory). They're two different use-cases, two different kinds of user.

[patrick91]

Hi there, sorry for bumping this issue, but I was wondering if there’s any update on this, I’d love to be able to trigger some leds when something happens on my computer. Didn’t find anything on google unfortunately

[adamhp]

@patrick91 Did you ever find a solution for host controlled colors?

[lukescott]

6829 was closed by the bot as stale because the discussion tag was removed. I would very much like to see some more discovery into this. In particular, integrating QMK keyboards into "Works with Razer" to sync lighting effects with games that utilize it (Overwatch, etc). Keyboard profiles for different apps would be another use-case, similar to big brand keyboard driver packages. Could go as far as creating a common QMK driver app for custom keyboards (optional).

[palp]

Coincidentally, over the weekend I built a working proof of concept for simple, fully host-controlled colors (not programming) on my Massdrop ALT through RAW HID. I piggybacked on @Wilba6582's work inside the VIA protocol, but with the intent of keeping it mostly distinct - it doesn't require VIA to be enabled. The usage of the protocol just boils down to a couple magic bytes on the messages sent over HID.

https://github.com/palp/qmk_firmware/tree/massdrop_rgbmatrix_hid

My initial test case is using Aurora as it has prior art for HID color control I could quickly iterate on. It's working and animations seem pretty smooth, but the keyboard hangs frequently. My embedded/C is pretty rusty; I have little to no error checking in the firmware, nor is it especially performant, so I expect that can be fixed. Hopefully in the near future I'll have time to debug that and polish things up a bit. It shouldn't be a huge leap to make it work with any keyboard supporting RGB Matrix. Building caches to look up LED indexes by keycode is probably the trickiest thing there. Aurora, and I imagine a lot of software, expects to address the LEDs by keycode. Addressing the RGBs by index directly from the host is better, but that would require a framework for the host to retrieve the current key mappings and get updates if they change - I use indexes only for the edge/underglow LEDs and the FN key for now. One line of thinking I like here is using the response data - anything sent with a keycode gets an LED index swapped in if applicable, and vice versa, so the host could invalidate a cache if it notices a change.

Here's a happy screenshot of my lit-up layout in Aurora:

[lukescott]

@pabile

Do the animation changes get sent in bulk via id_qmk_rgbmatrix_color_array with Aurora? The only other thing that stands out to me is [RGBK, ...] where K I'm assuming is keycode? So it looks up the keycode for each color change, right?

One way to speed it up would be some sort of lookup table, although I don't know what the memory is like on keyboards to support that. Another way to optimize this is by changing the data format to something like [K, ...] [RGB, ...]. This would allow the lookup to be done once in a couple cases:

• Multiple keys changing to the same color or animated together
• Single key with a complex animation (multiple colors)

I also don't see a delay between color changes. So perhaps that could be [K, ...] [RGBW, ...] where W is a delay in milliseconds.

The freezing could be caused by doing too much too quickly. While looking up keycodes might not be a problem by itself, maybe doing it for each and every color change with no delay in-between could be.

Another question I have is what happens to the layer-managed colors? I imagine, at least in the case of "Works by Razer", you'd want some sort of mode to allow the keyboard colors to be controlled by the game, but then change it back to the layer-managed colors afterward.

I have a Moonlander keyboard. I may be able to spend some time on this next week as I have some vacation time coming up. If not then, I have a week in December.

[palp]

@lukescott

Do the animation changes get sent in bulk via id_qmk_rgbmatrix_color_array with Aurora? The only other thing that stands out to me is [RGBK, ...] where K I'm assuming is keycode? So it looks up the keycode for each color change, right?

Not really in bulk. The first attempt I made was one-shot - send a commend to set X key to Y color. This worked, but performance was just awful. So, I more or less copied a pattern from the ASUS drivers in Aurora to queue updates and execute them per-tick at a given framerate (currently 30fps). HID reports are limited to (at most) 64 bytes, so the most I'll pack in to a single update is 16 (4 bytes each + 3 bytes for the header). However, when I flush the queue I create and send as many reports as needed, ignoring if the device can keep up or not. This solution is likely far from optimal, and was more just about getting something to work well enough that I could test and iterate.

The Aurora interface for devices takes a map of keycodes to colors, which could represent anything from 1 key to the entire device. It can also supply a bitmap, but I haven't dug too far into that - I'd likely be duplicating the work it does internally to map that, and doing so on device isn't useful. However, in the future a one-shot update that configures the entire RGB matrix instead of directly updating individual lights seems like a good idea. More on that below.

One way to speed it up would be some sort of lookup table, although I don't know what the memory is like on keyboards to support that.

Yeah, I wasn't really sure what the situation was like here, it's going to be a tradeoff on memory vs processing time, and the capabilities there are going to vary by keyboard so some implementation detail would likely be hardware dependent. This is why I was thinking of trying to cache those lookups on the host side and sending an LED index instead of keycode - not all LEDs have keycodes, either.

I also don't see a delay between color changes. So perhaps that could be [K, ...] [RGBW, ...] where W is a delay in milliseconds.

This is getting into territory Aurora doesn't go to - but it's an interesting idea. Right now RGB matrix is effectively disabled by setting LED_FLAG_NONE so it performs no work. This is by far the most compatible choice, as it doesn't require a framebuffer on device, but I think the better approach here may be creating a new mode that uses one. Extra work could be avoided, like setting a LED to a color it already is, and more complex operations like delays or patterns could be supported. Modes could even be added that coexist with and supplement direct control. However, this is going to be more limited by device memory/processing then directly setting colors, I'd guess.

Another question I have is what happens to the layer-managed colors? I imagine, at least in the case of "Works by Razer", you'd want some sort of mode to allow the keyboard colors to be controlled by the game, but then change it back to the layer-managed colors afterward.

At the moment I just don't care, but I think this is pretty easy to address. The driver would initialize the device, which could trigger storing the current mode/flags/framebuffer somewhere to be restored when control ends.

I have a Moonlander keyboard. I may be able to spend some time on this next week as I have some vacation time coming up. If not then, I have a week in December.

Great, I've just started to think more on cross-device compatibility but I think it'll be fairly easy with the basic functionality I have now. I may try to shift much of the code away from being keyboard specific by then, but no promises.

[palp]

A running theme here, unsurprisingly, is that getting much more sophisticated is going to introduce more device-dependent limitations. Even what I have is built around a specific implementation detail - the arm_atsam in the ALT uses a custom RGB matrix driver that is actually an extended LED Matrix driver. This is why I built it to use matrix - it was my only real choice that wasn't going to be device driver specific. I think the functionality of the RGB Matrix system will be good to have overall, but compatibility could be increased to include systems that only support lighting and not matrix. I don't know how interaction between lighting and the remotely controlled matrix would go since I don't have a RGB Lighting driver to use, but I assume some tweaking will be needed.

Further developments like using a framebuffer and restoring keyboard state are likely to not work as well on some devices as others, so I'd like to aim to degrade as gracefully as possible to provide a unified interface for as many QMK devices as possible. The host-side driver could compensate to fill in gaps where hardware support is limited, like keeping a keymap on the host to send LED ids, or saving/restoring the RGB matrix state. This would allow it to present a fairly unified interface for things like Razor Chroma that doesn't reduce functionality.

I need to do some research on the Razor SDK as well as others (OpenRGB is on my radar too) and figure out what functionality we need. Part of the reason I'm working with Aurora at first is that it's a very minimal implementation, as mentioned above - no programmability, delays, effects, etc - just control.

[palp]

Hah, it turns out there's some very recent work going on here (https://github.com/qmk/qmk_firmware/pull/10961) to support OpenRGB with quite a bit of overlap. At a glance it looks like it's doing it by LED index and supplying a (hardcoded) map to OpenRGB. This will work better in some situations, worse or not at all in others. It also is using a custom RGB matrix mode instead of directly setting the LEDs as data comes in, and supports other functions like changing modes. The support for processing multiple changes is different, too - it supplies a starting LED index and number of LEDs to change, then color data for each LED in sequence. Again, this would work better in some places and worse in others - it can be a more efficient use of the available data space, but only if you have the ability/desire to update the LEDs in sequence, not arbitrarily.

I think efforts here could and should be combined, but I would prefer to see it as a QMK specific protocol, not an OpenRGB one, as integration with any and all platforms ought to be the goal.

Associated branch on OpenRGB: https://gitlab.com/CalcProgrammer1/OpenRGB/-/tree/qmk_testing/Controllers/QMKRGBMatrixController

[lukescott]

I think using key code on the host side is a rather sensible approach given the host OS does not know or care if a keyboard has a standard layout or not. For example, in a game like Overwatch it knows which key codes bound to controlling a character, and it highlights those keys with specific colors that make sense for the game. The three most notable effects are single key solid color, single key pulsating on and off, and a wave effect in a specific color that goes across the entire keyboard. I have yet to dive into the Razer SDK to see how that is accomplished though.

[palp]

Reading up on the USB HID spec, I found this fairly recent addition: https://www.usb.org/sites/default/files/hutrr84_-_lighting_and_illumination_page.pdf

It'll certainly be a bit of a challenge to really dig in to HID to implement this, but since there's a spec, my inclination is to build to it. This can't share the RAW HID endpoint as it stands, unfortunately, as RAW uses report ID = 0 and that precludes any other HID reports existing in the descriptor. However, it could exist on the current shared endpoint, though I'm thinking making a new one intended for lower priority, higher throughput, two way communications would be better. RAW HID could be reimplemented on this with report ID 1, and other than losing a byte from MAX_EPSIZE it should be backwards compatible, in theory.

No idea if I'll make much headway on that soon, I'm still digging through the HID code and the spec. I did fix the crashing on my current implementation, so I have a functional reference point.

[lukescott]

Do all keyboards support this?

[palp]

Do all keyboards support this?

Hah, no, as it turns out very little if anything does, yet. I've got the HID device working save a bug with reporting mapped keycodes, but the result in Windows 10 seems to be that the OS creates its own connection with the device and locks out raw HID connections to that interface. You can get at it through the UWP Windows API, which is neat, but not as useful as I'd like without building my own full on app. If I change the usage page, though, it doesn't pick it up and I can talk to it normally. So leaving that user-definable lets you choose how the interface will work. The Windows driver layer does seem to do a nice abstraction, so I may play with building a host-side API bridge using that, which could potentially link in to Razer, AURA, etc. Though, the idea, hopefully, is that other standards add interop with this one since it's built in to Windows 10.

Here's what I'm working off so far. I'm trying to pull as much code into my keymap as possible for now, adding it as a keyboard specific feature seems like a good first step.