Add specification for AltGraph key & modifier behaviour

[drwez]

The UI Events specification caters for a variety of modifier keys, including AltGraph (also known as ISO Level 3 Shift). In general how modifiers (e.g. Shift, Control) affect input is fairly consistent and well understood across platforms.

AltGraph is a more complex case on some platforms, and user agents typically expose the per-platform behaviors to web content. Under Windows, for example, there are four main complications for content authors:

1. Some user agents report keydown/keyup events with KeyboardEvent.key = "Alt" rather than "AltGraph". Since the meaning of the right-hand Alt key depends on the keyboard layout in effect, this may cause sites with specialist keyboard handling to mis-interpret AltGraph as Alt.

2. All user agents tested (Chrome, Firefox, Edge) report AltGraph-modified keydown/keyup events with the Control and Alt modifiers also set. Although this accurately reflects how Windows represents AltGraph in input events, it creates confusion for content which associates special meanings with specific Control+ key combinations, if is reached via AltGraph under the current layout.

3. Some user agents report the AltGraph modifier on events whenever both Control and Alt are active, even under layouts which do not use AltGraph at all.

4. Some user agents generate keydown/keyup events with a KeyboardEvent.key value which does not represent the character actually generated by an AltGraph+ sequence.

5. All user agents tested (Chrome, Firefox, Edge) directly reflect the underlying platform key event sequence for AltGraph press/release, generating keydown with code=ControlLeft, keydown code=AltRight rather than a single keydown code=AltRight event.

We should at least provide an informative section in the specification to illustrate the issues that content authors must allow for, and ideally provide a normative specification for the expected behavior user agents should arrange to provide, where feasible.

+@garykac

[masayuki-nakano]

I think that ideally, when AltGraph is pressed on Windows, neither Ctrl nor Alt should not be exposed because the modifier's meaning has been changed.

If AltGr + KeyA is pressed, following events are fired (perhaps):

1. LeftCtrl or RightCtrl is down
2. LeftAlt or RightAlt is down
3. KeyA is down and up
4. LeftCtrl or RightCtrl is up
5. LeftAlt or RightAlt is up

(1 and 2, 4 and 5 may be swapped)

Then, events at 1 cannot be exposed as AltGr since only one modifier is active at this time. 5 is similar to it. Next, 2, 3 and 4 can be AltGr. However, I have no idea how to distinguish if the keyboard layout has AltGr function.

On the other hand, there is a possible issue. For example, if the events are fired with following order:

1. CtrlLeft keydown
2. AltRight keydown
3. CtrlLeft keyup
4. AltRight keyup

If 2 and 3 don't have raw modifier information, i.e., "Alt" nor "Ctrl", web apps may failed to manage modifier state. E.g., if a web app entering special mode only while Ctrl key is pressed, pressing both Alt and Ctrl keys may cause locking it into the mode if 3 is treated as "AltGr" keyup event.

However, I have no idea of problem if browsers remove Ctrl and Alt modifier state from the events between 2 and 4 in the first example.

FYI: In Japan, most users never use AltGr key. Therefore, some compact keyboards don't have AltRight key (and also MetaRight). So, I sometimes need to use both CtrlLeft and AltLeft to test some Western keyboard layouts.

[masayuki-nakano]

On the other hand, on macOS, it has different problem.

Option key is typically treated as Alt key if it's used as shortcut key or something. However, it actually works as AltGraph key when user types text. So, it seems that including both Alt and AltGraph modifier state on macOS may makes sense.

[drwez]

Re modifier flags: I've prototyped code to disambiguate AltGr from Ctrl+Alt - I was concerned about the issue of confusion between Ctrl followed by Alt, and a single AltGr, and about the risk of content which actually needs to "see" Ctrl+Alt no longer being usable. The prototype basically:

• Scans the keyboard layout to establish whether it uses AltGraph (i.e. does Ctrl+Alt+ generate a character for any value of ?).
• Iff layout has AltGraph then:
  • Maintain an internal flag, ignoreLeftControl.
  • Set that flag depending on the precise order of Control and Alt WM_KEYDOWN events received.

This approach treats Ctrl+Alt as Ctrl+Alt, and only the specific sequence of keydown-ControlLeft followed by keydown-AltRight triggers AltGraph. The user can still Ctrl-modify an AltGraph-modified key by pressing e.g. ControlLeft after pressing AltGraph - we can spot that case because Windows sends the WM_KEYDOWN event even though the key is logically already "down".

This prototype works for me, but the state-machine is fiddly, so we should definitely consider more straightforward options, such as always treating Ctrl+Alt as AltGraph, if we're confident they won't break content on Windows.

Re the event sequence: It's possible to "hold" the keydown-ControlLeft and not dispatch it to content if it turns out to be AltGr, as part of the logic I've prototyped (see above), but as well as adding complexity, is there a risk that existing content relies on "seeing" both events for AltGr?

[drwez]

And yes, it looks like MacOS uses Option as Alt or AltGraph on a per-key basis, under layouts with AltGraph-generated keys? We can still have the AltGraph modifier set correctly on events, but I don't think it will be possible to generate keydown/keyup events for Option itself, with the correct KeyboardEvent.key values?

[drwez]

Oh, I missed your comment regarding compact keyboards in Japan not having the AltRight/AltGr key - that's good to know. Do you know offhand how Linux distributions handle that, given that they don't have the Ctrl+Alt fall-back for AltGr?

[masayuki-nakano]

Scans the keyboard layout to establish whether it uses AltGraph (i.e. does Ctrl+Alt+ generate a character for any value of ?).

I don't think that it's good idea due to performance reason. Although if doing it makes web developers really happy, we could take the approach.

Re the event sequence: It's possible to "hold" the keydown-ControlLeft and not dispatch it to content if it turns out to be AltGr, as part of the logic I've prototyped (see above), but as well as adding complexity, is there a risk that existing content relies on "seeing" both events for AltGr?

I don't think that Browsers should put off to dispatch Ctrl or Alt key down. E.g., if user Ctrl or Alt is down, then, mouse move, finally, other one is pressed, how should browsers do? Only optimizing for modifier keydown event doesn't make sense to me.

We can still have the AltGraph modifier set correctly on events, but I don't think it will be possible to generate keydown/keyup events for Option itself, with the correct KeyboardEvent.key values?

Yes, but I don't think that it's important to fire "AltGr" keydown event because most web apps don't need to listen modifier keydown events. If some web apps need to do that, they can do it with listening to all keydown events and check getModifierState("AltGraph").

Do you know offhand how Linux distributions handle that, given that they don't have the Ctrl+Alt fall-back for AltGr?

Looks like that it depends on user settings. At least, Ubuntu has GUI to set specifying "AltGr" key for Level3 and Level 5 to a key or a key combination. For example, user can set CapsLock to AltGr, Backslash to AltGr, Menu ("ContextMenu") to AltGr, etc. So, cannot do any hack for modifier keydown on Linux.

I believe that if web apps need to handle AltGr keydown and keyup, they should do:

var isAltGrDown = false;
foo.addEventListener("keydown", (event)=>{
  isAltGrDown = event.getModifierState("AltGraph");
});
foo.addEventListener("keyup"), (event)=>{
  isAltGrDown = event.getModifierState("AltGraph");
}); 

[drwez]

Thanks for the input, Masayuki, and apologies for the slow response.

It sounds like we would be in agreement on the following, then:

1. Add normative text explaining that content should expect that keydown/keyup events for AltGr may reflect the platform-dependent mechanisms used to access the extra characters.
2. Specify that under keyboard layouts with AltGr-shifted characters on one or more keys, if the platform accesses those characters via an existing non-AltGr modifier/combination then that modifier/combination should be cleared on the keydown/keypress/keyup events, and the AltGr modifier set instead.
3. Specify that under non-AltGr layouts, or for keys which do not generate any character under the platform-specific AltGr modifier/combination, the platform-specific modifier/combination flags should be reported, rather than replacing them with AltGr.

This should meet the following requirements: a. Content can distinguish AltGr-generated characters from Ctrl/Alt modified keys on Windows. b. Users can still enter AltGr-shifted characters via Control+Alt+ if necessary. c. Users can still access Control+Alt modified sequences, even under AltGr layouts, under Windows, provided the modified would not generate a printable character.

WDYT?

I still think it would be valuable for UAs to have the option of "fixing" the keydown/keyup event sequence, but that is definitely non-trivial, and mostly serves specialist content (e.g. remote-access clients), so seems less critical to address.

Regarding "scanning" the layout, Chrome already does this whenever the active layout changes, to collate a lookup table from which to generate |key| values for subsequent events, FWIW.

[masayuki-nakano]

Add normative text explaining that content should expect that keydown/keyup events for AltGr may reflect the platform-dependent mechanisms used to access the extra characters.

Hmm, I don't understand what you meant. AltGraph keydown/keyup events may not be fired with platform limitation (e.g., macOS), but AltGraph state could be set to InputEvents.

Specify that under keyboard layouts with AltGr-shifted characters on one or more keys, if the platform accesses those characters via an existing non-AltGr modifier/combination then that modifier/combination should be cleared on the keydown/keypress/keyup events, and the AltGr modifier set instead.

I don't understand this well. Did you mean that even when Ctrl+Alt is pressed on Windows with keyboard layouts which has AltGr key, then, Control and Alt modifiers should be cleared and AltGraph is set? If so, I agree with that.

Specify that under non-AltGr layouts, or for keys which do not generate any character under the platform-specific AltGr modifier/combination, the platform-specific modifier/combination flags should be reported, rather than replacing them with AltGr.

Hmm, so, did you mean that if AltGr key doesn't exist as an independent key (like Windows), only when KeyboardEvent inputs one or more characters, AltGraph should be set but otherwise, should be set only actual modifiers (e.g., Ctrl + Alt)? If so, I agree with that. However, it'll never set AltGraph state to other InputEvents like MouseEvent on Windows. But I have no idea how that causes problem for web developers.

This should meet the following requirements: a. Content can distinguish AltGr-generated characters from Ctrl/Alt modified keys on Windows. b. Users can still enter AltGr-shifted characters via Control+Alt+ if necessary. c. Users can still access Control+Alt modified sequences, even under AltGr layouts, under Windows, provided the modified would not generate a printable character.

Agree.

Regarding "scanning" the layout, Chrome already does this whenever the active layout changes, to collate a lookup table from which to generate |key| values for subsequent events, FWIW.

Gecko does it only on Windows but not so on the other platforms. I don't know the performance of macOS and GTK is enough fast. I have no idea about Android since it provides virtual keyboard with IME.

[drwez]

On 20 September 2017 at 03:06, Masayuki Nakano notifications@github.com wrote:

Add normative text explaining that content should expect that keydown/keyup events for AltGr may reflect the platform-dependent mechanisms used to access the extra characters.

Hmm, I don't understand what you meant. AltGraph keydown/keyup events may not be fired with platform limitation (e.g., macOS), but AltGraph state could be set to InputEvents.

My point is that the spec should explicitly mention that the applications cannot assume a simple sequence of keydown/keyup events for AltGraph, in general, across all platforms.

Specify that under keyboard layouts with AltGr-shifted characters on one or more keys, if the platform accesses those characters via an existing non-AltGr modifier/combination then that modifier/combination should be cleared on the keydown/keypress/keyup events, and the AltGr modifier set instead.

I don't understand this well. Did you mean that even when Ctrl+Alt is pressed on Windows with keyboard layouts which has AltGr key, then, Control and Alt modifiers should be cleared and AltGraph is set? If so, I agree with that.

If the Ctrl+Alt+ sequence leads to a printable character being generated, then Ctrl+Alt should be cleared, and AltGr set, so that content can "see" that the character was generated via AltGr.

If Ctrl+Alt+ does not lead to a printable character then leave Ctrl+Alt modifiers set in the event.

Specify that under non-AltGr layouts, or for keys which do not generate any character under the platform-specific AltGr modifier/combination, the platform-specific modifier/combination flags should be reported, rather than replacing them with AltGr.

Hmm, so, did you mean that if AltGr key doesn't exist as an independent key (like Windows), only when KeyboardEvent inputs one or more characters, AltGraph should be set but otherwise, should be set only actual modifiers (e.g., Ctrl + Alt)? If so, I agree with that. However, it'll never set AltGraph state to other InputEvents like MouseEvent on Windows. But I have no idea how that causes problem for web developers.

Yes, that would be a down-side of this approach. Remember that if content actually expects an AltGr modifier to be set on those events then it is out of luck on one of the reduce-size keyboards you mentioned, that don't actually have a physical AltGr key.

An alternative would be to allow both explicit AltGr key handling and Ctrl+Alt+ that generates a printable key, so that:

1. If you press AltGr on a keyboard with a physical AltGr key then the UA should set the AltGr modifier, and clear any platform-specific modifiers like Ctrl+Alt, on subsequent events.
2. If you press Ctrl+Alt (or some other platform-specific combination) to mimic AltGr, and the result is a printable character then the keydown/keypress/keyup for the character key should have AltGr, rather than the other modifiers.

This would allow UAs to support use-cases served by AltGr as a general modifier, while also allowing it to work sensibly when synthesized e.g. via Ctrl+Alt.

This should meet the following requirements: a. Content can distinguish AltGr-generated characters from Ctrl/Alt modified keys on Windows. b. Users can still enter AltGr-shifted characters via Control+Alt+ if necessary. c. Users can still access Control+Alt modified sequences, even under AltGr layouts, under Windows, provided the modified would not generate a printable character.

Agree.

Regarding "scanning" the layout, Chrome already does this whenever the active layout changes, to collate a lookup table from which to generate |key| values for subsequent events, FWIW.

Gecko does it only on Windows but not so on the other platforms. I don't know the performance of macOS and GTK is enough fast. I have no idea about Android since it provides virtual keyboard with IME.

We've been focused on the Windows case in our implementation so far; performance for that seems to be OK, since we're only scanning the keyboard on the first input event after a layout switch. But yes, we should avoid requiring behaviour that can't be implemented without potentially expensive scanning processes.

For platforms which usually run with IME input, I'm not sure that this problem applies - the extended Latin-1 characters, for example, will typically be accessed directly via long-press in the IME rather than via setting a virtual modifier.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/w3c/uievents/issues/147#issuecomment-330807122, or mute the thread https://github.com/notifications/unsubscribe-auth/AcZxkrQzRFXTkj9Mwc96ObCWdsvqxgi2ks5skOOzgaJpZM4OKABk .

[drwez]

Masayuki,

FYI, I've uploaded a draft implementation of this proposal, for Chromium on Windows, at https://chromium-review.googlesource.com/c/chromium/src/+/558584

If this seems reasonable (especially the balance between getting the behaviour right, and the implementation complexity) then I'll work with garykac@ to get some spec-friendly wording for review.

[masayuki-nakano]

Thank you, I'm not sure if I fully understand the code of Chromium, though. But looks fine to me. (I completely agree with your previous answers. Sorry for the delay to reply.)

Anyway, I'd like to check Chromium's behavior and feedback. So, I'd be happy if you'd let me know when you'd the change.

[drwez]

Masayuki, we've implemented most of the proposed behaviour behind a "feature flag" in Chrome; you can grab Chrome Canary and run it with --enable-features=FixAltGraph to see the new behaviour.

Currently things work as intended for the physical AltGr key; all events are tagged with AltGraph set, rather than Control and Alt, while AltGr is down. keydown events generated while Control+Alt are also correctly switched to AltGraph, if the key is AltGraph-modified, but keypress events still report Control+Alt right now.

[drwez]

We now have this working as described for keydown, keypress and keyup events, from tomorrow's Chrome Canary build. You can run the build w/ the command-line option "--enable-features=FixAltGraph" to try the new behaviour.

@masayuki-nakano Could you try this out and see what you think?

[masayuki-nakano]

Oh, sorry for the delay to reply. Yes, I'll try to check it but I might not have much time to work on changing Gecko's behavior. But I'll make time to work on it if you enable it in default settings. When are you gonna release it in the default settings?

[drwez]

@masayuki-nakano You can enable it in Chrome Canary via --enable-features=FixAltGraph; by "default settings" do you mean allowing it to be set via about:flags, or actually having it enabled by-default for all users?

[masayuki-nakano]

@drwez I want to know the last thing. I.e., when you'll enable it by-default for all users.

[RByers]

Ping @patrickkettner for any feedback from Microsoft. We're discussing shipping this change in chromium here.

[masayuki-nakano]

Filed a bug for Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1444581 And I'll start to work on it soon. However, I'm still not sure when we should use the new behavior in release channel.

[patrickkettner]

The proposal looks solid. I looked over it along with the input team and we don't see any issues.

+1

[drwez]

@masayuki-nakano I reviewed https://bugzilla.mozilla.org/show_bug.cgi?id=1444581 that you filed, which seemed to only describe the |key| value for keydown/keyup events from the key with |code=AltRight|. The behaviour we were discussing on this bug is which modifiers are listed by getModifierState() - in particular that the user agent should not report "Alt" and "Control" modifiers where either AltRight or AltLeft+Control are being used to set the AltGraph modifier.

[masayuki-nakano]

Thanks for the review and notifying me of the points.

The bug report doesn't mention everything, though. Firefox already doesn't send Alt nor Control state for keypress events which will cause text input. So, we can adjust the behavior for keydown (and keypress for non-printable key cases) easily. Perhaps, it's so for keyup too.

[masayuki-nakano]

FYI: Due to our design of the implementation, it's difficult to change the behavior with a pref. So, when I fix the bug, it'll ride the train immediately.

[gked]

@drwez Could you please elaborate on what the last constrain mean in the algorithm defined here - https://chromium-review.googlesource.com/c/chromium/src/+/558584 ?

-Whenever AltGraph is set, Control & Alt are cleared, and vice versa.

[drwez]

Basically the UA should report either that the event is AltGr-modified, or Control+Alt-modified, but not both, since under Windows the two meanings cannot be disambiguated, in general.

[masayuki-nakano]

@drwez Hi, I wrote patches for https://bugzilla.mozilla.org/show_bug.cgi?id=900750 . The summary of the new behavior is at comment 24.

As far as I've tested with Chrome without --enable-features=FixAltGraph, looks like that Chrome has already started to use "AltGraph" key value for AltRight key event when active keyboard layout has AltGr key. On the other hand, looks like Chrome still sets altKey and ctrlKey to true when getModifierState("AltGraph") returns true. I see this on an environment but I don't see such behavior on another environment. Do you do A/B test now?

Additionally, as I guessed, it's difficult to make this behavior change behind pref. So, we need to wait to land the patches until Chrome enables the new behavior in release channel. So, I'd like to know when you enable it in release channel by default.

[drwez]

@masayuki-nakano we provided the FixAltGraph feature setting in Chrome M67, and switched in on permanently in M68. If you test with M68, you should always see the new behaviour.

[masayuki-nakano]

Oh, thank you very much. I checked the version of Chrome which still sets ctrlKey and altKey to true, then, it starts update. The version was 66. And indeed, starting from 67, those modifiers are cleared now. Thank you for the information!

[masayuki-nakano]

FYI: I landed the fix for Firefox 63 Nightly. Firefox 63 will be released on 2018-10-23.