Support for Media Keys

[pete-otaqui]

Media keys that exist on some keyboards, for example "Play/Pause", "Next Track" and "Previous Track could be handled by TinyPilot.

In my case, I am actually using TinyPilot to remote control a variety of media devices. These are usually managed with a bluetooth / IR remote control, or other more-exotic-than-a-keyboard control device, but generally do have support for a keyboard being plugged in. There are a couple of types that only provide full functionality via media keys when using a physical keyboard.

HID Codes

I haven't verified these, but I found a reference to the HID codes with this information:

HID CODE    | Key
------------|------------------------
0x07 0x00e8 | Media Keys Play/Pause
0x07 0x00e9 | Media Keys Stop
0x07 0x00ea | Media Keys Previous
0x07 0x00eb | Media Keys Next
0x0c 0x00b3 | Media Keys Fast Forward
0x0c 0x00b4 | Media Keys Rewind

I notice that you already have the Play/Pause HID code in keycodes.py, however I'm not sure how it would be used since the virtual keyboard does not have this key, and I don't believe the UI will capture it and forward it on - please do let me know if I'm wrong about that.

Physical and Virtual Keyboard

Capturing these key strokes from a physical keyboard in a browser is different to normal keypresses. Instead of addEventListener, one must use the MediaSessionApi.

In any case, it seems that the couple of browsers I tested (Firefox and Chrome on macOS) ignore these action handlers if there is no "active" media session (see note*)

That being said, it would obviously be possible to add media keys to the virtual keyboard.

Updated socket API

The "real" callback arguments for the action handlers in the MediaSessionApi are not like normal key events, but much more simplistic:

navigator.mediaSession.actionHandler('play', (arg) => console.log(arg));
// the above outputs `{ action: 'play' }`

Between the different event types, and the different callback arguments, it seems as though it would be wrong to extend the existing keystroke socket API with "fake" media key data. An alternative option would be to introduce a new part to the socket api, specifically for this purpose.

Ultimately pressing these keys could lead to emitting something like the following:

socket.emit('mediaKeystroke', { action: 'play' })

There would then presumably need to be an equivalent request_parsers/media_keystroke.py - this file could be simpler than the existing keystroke.py parser, since it would not have to handle any modifiers. I guess that js_to_hid.py would also need a convert_media(mediaKeystroke) similar to the existing convert(keystroke). And some other updates, most notably to socket_api.py to implement the rest of the parts.

--

*note If I load a youtube video in the browser, the "Play/Pause" button will work to control the video. If I then use the developer console to set my own action handler, it takes over and my handler is called instead of controlling the video when I press the button. However - if I add the same handler on an arbitrary non-video-containing page it does nothing at all.

[mtlynch]

Sorry for missing this! I typed up a response a few days after you created the issue, but I must have accidentally closed the tab without publishing it.

It's a bit of a hectic week, but I should be able to respond to this next week.

[mtlynch]

Sorry again for the delay on this @pete-otaqui!

I actually get keystroke events for the media buttons on my keyboard in any window on Chrome and Firefox on Win10:

document.addEventListener("keydown", (evt) => {console.log(evt);});

Chrome

VM120:1 KeyboardEvent {isTrusted: true, key: "AudioVolumeMute", code: "", location: 0, ctrlKey: false, …}
VM120:1 KeyboardEvent {isTrusted: true, key: "AudioVolumeDown", code: "", location: 0, ctrlKey: false, …}
VM120:1 KeyboardEvent {isTrusted: true, key: "AudioVolumeUp", code: "", location: 0, ctrlKey: false, …}
VM120:1 KeyboardEvent {isTrusted: true, key: "MediaPlayPause", code: "", location: 0, ctrlKey: false, …}
VM120:1 KeyboardEvent {isTrusted: true, key: "MediaPlayPause", code: "", location: 0, ctrlKey: false, …}

Firefox

keydown { target: body.illustrated, key: "MediaPlayPause", charCode: 0, keyCode: 0 }
keydown { target: body.illustrated, key: "AudioVolumeDown", charCode: 0, keyCode: 182 }
keydown { target: body.illustrated, key: "AudioVolumeUp", charCode: 0, keyCode: 183 }
keydown { target: body.illustrated, key: "AudioVolumeMute", charCode: 0, keyCode: 181 }
keydown { target: body.illustrated, key: "AudioVolumeMute", charCode: 0, keyCode: 181 }

Is behavior different on other OSes? This might be as simple as adding mappings to js_to_hid.py.

[pete-otaqui]

Hi, apologies for the delay in replying.

Yes, the behaviour is different in macOS - media key presses are not captured at all by document.addEventListener.

Perhaps a good solution is to provide media keys in the virtual keyboard?