Consider addition of a property for input controls to provide untranslated Unicode Braille Patterns (U+2800..U+28FF)

[samdooley]

This issue is being posted there at Joanie's request as a place to start the discussion around how to provide untranslated Unicode braille input codes from a screen reader to a web application. We're not saying that this should necessarily be an ARIA feature, but it seemed that this would be a good place to start the discussion.

Example: A math editor has a visual user interface that is designed to accept and interpret Unicode braille symbols produced by a braille device, and would like to distinguish input events for braille symbols from input events for ascii characters from a computer keyboard. The math editor is prepared to accept key events that include a property that would indicate a braille key press in the Unicode braille range U+2800 to U+28FF to indicate the braille key combination.

[jcsteh]

nvaccess/nvda#8340 proposes aria-braille-input. I wonder whether it might be better to have aria-text-input-type="braille" or similar. This would be akin to the type attribute on the HTML input element (but only for text types).

[jwmerrill]

As a concrete use case for this feature, the Desmos four function and scientific calculators are able to accept mathematical input and display mathematical output in either Nemeth or UEB.

• Intro video: http://learn.desmos.com/braille
• Calculator (to activate braille mode, open the settings "wrench" menu, and choose Nemeth or UEB): https://www.desmos.com/scientific
• Documentation about using Braille input: https://www.desmos.com/accessibility#braille

Currently, for this feature to work, we have to ask users to switch their input into "uncontracted 8-dot computer braille" (the terminology varies platform-to-platform). This is necessary because contractive Braille tables (which are very commonly used) will expand certain Braille cells as contractions and interfere with our ability to interpret the user input as Nemeth.

We have found that asking users to switch their input table can be a significant hurdle; especially for students and teachers who may not be expert screen reader users.

If we could hint to the screen reader that a particular input knows how to handle raw Braille characters, then users could enter Nemeth or UEB math into the calculator without having to switch their system-wide input table.

[clapierre]

This is very important for us at Benetech and the DIAGRAM Center as we are working currently on a web based digital math editor (called MathShare) and being able to have raw braille input into our editor will be crucial for those students using braille as their input modality into our editor. We have just received funding to continue to work on MathShare from Cisco Foundation and General Motors

[pkra]

Just to write down what I said on today's call before it started formally. I'd be happy to spec this out for 1.3 (alongside the other braille properties) but I really don't know enough about input. So if the group can figure out how this can work, I'm happy to write it up and iterate.

[pkra]

@jnurthen quick ping to put this on the agenda some time.

[pkra]

To gather two specific comments from an early prototype at https://github.com/nvaccess/nvda/pull/8340

@michaelDCurran wrote at https://github.com/nvaccess/nvda/pull/8340#issuecomment-392737982

We could in deed use a role. The ARIA workinggroup may well suggest that it should be an ARIA role rather than an ARIA property.

@jcsteh wrote at https://github.com/nvaccess/nvda/pull/8340#issuecomment-395636362

I'd want to see use cases for non-text inputs accepting raw braille. I understand that's how it's implemented in NVDA, but whether it's appropriate for such a low level concept to be supported by the spec is a different matter. Supporting such a low level concept is bordering on saying that semantics are irrelevant, and that's a dangerous precedent to set for ARIA.

FWIW, users can already choose to enter raw braille input into a text box, but they'd have to manually switch their input table. As I see this, it's just a hint to the screen reader saying "this text input accepts braille dots", comparable to "this text input accepts an email address".

Still, I guess this is a matter to debate at the spec level.

[pkra]

And in response to @jcsteh's request

I'd want to see use cases for non-text inputs accepting raw braille.

IIRC desmos and Pearson's AEE both use hacks to get raw braille input (of Nemeth Braille, I believe).

I can imagine similar use cases for chemical braille and braille music.

[LeonarddeR]

I think I agree with @jcsteh when he says in https://github.com/w3c/aria/issues/771#issuecomment-395630902:

I wonder whether it might be better to have aria-text-input-type="braille" or similar.

Also, if it is merely a hint to the screen reader that it should provide unicode braille as input, how are we supposed to ensure that no other characters are ending up in the text control? I assume the web author is supposed to deal with that, but it is easy to overlook such a thing. There should probably be a best practice that gives directions to do this.

[carmacleod]

Sina's example: https://sinabahram.github.io/aria-playground/BrailleInput.html

[cookiecrook]

There are probably several ways this could be done, but after some brainstorming today, here's one possibility.

<!-- 
Notes: 
- ⠉⠕⠎ is "cos" short for cosine in Nemeth Math braille. Trying to keep the example short. 
- The field label "formula" is intended to explicitly show that that the label attrs 
  should not be confused with or overloaded with values.
-->
<input aria-label="formula" aria-brailleinput="nemeth" value="⠉⠕⠎">

<!-- optionally with another aria-speechvalue attr if the app 
     wants to handle the explicit back translation -->
<input aria-label="formula" aria-speechvalue="cosine" aria-brailleinput="nemeth" value="⠉⠕⠎">

1. Two new attributes: I'm not partial to the bikeshed names here being aria-speechvalue and aria-brailleinput

2. aria-brailleinput any value besides empty/none/undefined, such as "nemeth" or another unambiguous braille table identifier (recognized or not for future compat) would mean the native input or contenteditable input would not be translated on input or output, and the app should limit the field value to unicode braille range. Brailled characters would be input directly as unicode equivalents. The fields would not be able to intercept braille chords (commands), as those are reserved for the screen reader. A recognized value (e.g. "nemeth") may allow the screen reader to perform back translation to speech, which could then be delivered to the text-to-speech engine.

3. Another optional (or perhaps required) attribute is aria-speechvalue. Given that the value is limited to braille unicode and the back translation may be ambiguous, it could allow web apps to offer a speech string.

Privacy:

• There is still the outstanding issue of inadvertently "outing" a braille SR user.
• Besides an SR, the only way I'm aware of to insert braille directly is to paste in unicode chars, or insert them via the character viewer palette (Ctrl+Cmd+Space on Mac, then search for a specific dot pattern, like "135")
• The speech value attr may need to be be limited to the contexts where the field value is braille, to prevent malicious web apps from masking sensitive input values from an AT user.
• No raw braille key events (e.g. keydown) should be sent to the web application.

[cookiecrook]

Currently this issue is planned for broader (but still informal) discussion in the ARIA Deep Dive call on December 10, 2020 at 9 AM Pacific. This time uses the same IRC and Zoom channel as the ARIA WG meeting, but is one hour earlier.

[sinabahram]

Thanks for this @cookiecrook . A meta issue first, if I may.

I fear this is a lot of potential scope creep, and I'm concerned all of the above phrases this issue in such a way that AT vendors will not want to implement it due to it being a lot of work.

I suppose what I was trying to convey on the call was simply an element with no implications around speech or indicating translation tables or any of that. Simply a mechanism for the screen reader to pass through the raw braille dots entered into an edit field. Everything else needs to be their own respective issues it seems, and could be handled via application logic (though of course I would love for screen readers to help in that loop).

I'm happy to respond to the above in detail and have all those conversations around specifics, but I first wish to ask if you feel it appropriate to simplify the scope of this request because currently it has callouts to speech and such, whereas I was advocating for just raw braille dots to be passed in if this attribute is set to true.

If not, and you feel all of these issues need to be discussed together, then I'm delighted to dive in and begin responding.

Thank you again for the writeup, and I can't wait to begin figuring this out.

[cookiecrook]

I suppose what I was trying to convey on the call was simply an element with no implications around speech or indicating translation tables or any of that. Simply a mechanism for the screen reader to pass through the raw braille dots entered into an edit field.

That's what the combination of two attributes would allow. Requiring aria-speechvalue means the implementation of aria-brailleinput could be simplified... No back translation required.

The implementation would only become trickier if aria-speechvalue was optional, because then a speech-enabled screen reader would need to infer what to speak from the input's braille value.

[cookiecrook]

The December 10th discussion time is no longer going to work for the Apple engineers that wanted to attend. What about the following week, December 17th, or in January, if too many people will vacation early ahead of the December holiday?

[sinabahram]

Either works for me in terms of dates, though I am trying to wrap most work up that week with the 17th. This is super-important, though, so I'm happy to make time for it. I also owe you answers to the above comment, and I very much have not forgotten that/am planning on still doing so.

[cookiecrook]

I changed the name of the issue because, in discussion, no one has asked for raw braille key events. AT vendors would likely object to providing raw braille display key events anyway.

Instead, the need is to mark an input field as accepting untranslated unicode braille symbols, referred to in the Unicode spec as Braille Patterns (U+2800..U+28FF) such as dots124 (⠋), dots1234567 (⡿), dots12468 (⢫).

[samdooley]

Either date works for me as well.

Thanks for the clarifications.

Sam

On Thu, Dec 3, 2020, 11:33 AM James Craig notifications@github.com wrote:

The December 10th discussion time is no longer going to work for the Apple engineers that wanted to attend. What about the following week, December 17th, or in January, if too many people will vacation early ahead of the December holiday?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/w3c/aria/issues/771#issuecomment-738165483, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEIHUAQSH2RACUZHUFJZ63SS7DWFANCNFSM4FB2OXHA .

[sclower]

I can attend on December 17 to add in the perspective of what we've done so far at Desmos to support Braille input. I am glad some Apple engineers will be present to discuss this issue in more detail as we have had the most challenges making our Braille editor work consistently on iOS and the Mac.

[cookiecrook]

Thanks @sclower.

If you're saying that you've gotten untranslated braille input working on other systems, are you able to share more details how you achieved that in the absence of a standard?

If you're referring to other, unrelated braille issues, ping me offline or let's discuss those in other relevant issue threads.

[samdooley]

Thanks James.

Speaking for the Pearson equation editor, it requires the braille terminal user select ASCII braille output, which is sent to a hidden input control in the web page where the javascript intercepts the key events. The javascript editor provides a mode switch to allow the user to select between standard keyboard input, braille terminal input, and braille input using the home row keys on the standard keyboard. The input handler then knows how it should interpret the key events that are coming into the input control, and respond to the corresponding braille patterns as Nemeth input. However, it puts the burden on the user to make sure the input device being used matches the input mode expected by the software.

The idea is that if there were a way to have the braille terminal send U+28xx braille patterns, that could potentially simplify both the input logic and the user experience.

Sam

On Tue, Dec 8, 2020 at 1:31 AM James Craig notifications@github.com wrote:

Thanks @sclower https://github.com/sclower.

If you're saying that you've gotten untranslated braille input working on other systems, are you able to share more details how you achieved that in the absence of a standard?

If you're referring to other, unrelated braille issues, ping me offline or let's discuss those in other relevant issue threads.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/w3c/aria/issues/771#issuecomment-740438827, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEIHUATAKJK475IO7MVPSLSTXI4NANCNFSM4FB2OXHA .

[cookiecrook]

where the javascript intercepts the key events.

Ah okay, so the user types ASCII braille (e.g. ⠉⠕⠎) which is sent to the webpage as standard keyboard input (e.g. "cos"). Then the web app translates to Nemeth ⠉⠕⠎ and interprets as "cosine." Is that right?

[sclower]

Hi James,

On the Desmos side, we have added Nemeth and UEB I/O modes to our four-function, scientific, and graphing calculators. When either mode is active, we render standard text input fields which accept Braille ASCII. The ASCII is internally converted to the UNICODE dot patterns, and we have designed our Simbraille font to render the dot patterns for both ASCII and UNICODE versions of each character.

A six-key input mode is also available, and when active, we translate chorded input on the homewor keys into the expected Braille patterns.

Some compromises we have had to make in our design to best support Apple devices include:

1. We originally designed our Braille editor to input and output Braille UNICODE dot patterns by translating any newly entered ASCII into UNICODE as it was entered. This appeared to present no problems on Windows using JAWS, NVDA, and now Narrator. On Mac OS, however, the replacement would cause rapid flashing of characters on a connected Braille display which resolved to a cursor blinking on a blank line of text. We ultimately decided to adapt the editor to receive and output ASCII instead so that no transformation of entered content would occur, thus making it feasible for students with visual impairments to use it. This does also mean that if a screen reader's speech is active, users will hear the ASCII characters read aloud instead of the dot pattern descriptions.

2. In our typeset editor, we have added functionality to cause a screen reader to speak pertinent information as the user navigates through equations via an ARIA live region. We considered adapting this logic for our Braille editor to give speech and Braille users a fuller experience. Were we supporting only Windows, we could do this since neither JAWS nor NVDA send live region content updates to the Braille device; the region updates are communicated only through speech. This is also possible on the Mac if one disables the "show alerts for duration" option in the VoiceOver utility's Braille settings. On iOS, however, it is impossible to disable live region text from appearing in Braille, so this approach has been shelved for the time being.

Like Pearson, we require the user to disable any active literary translation on the user's device beforehand because there is no way to do so automatically. We have considered creating JAWS scripts or NVDA add-ons to handle this for Windows users, but again, lack of VoiceOver scripting makes this impossible on Apple devices. Though we have written documentation for disabling Braille translation for all of the platforms we support, it is still an obstacle for some, and may not be easily accomplished by those who are uncomfortable with (or may be barred from) changing screen reader settings (such as in the context of a locked machine during end-of-year exams). Then, too, the user must remember to switch the translator back on when he/she has finished working with the editor. A means by which an input could communicate the need for Braille to a screen reader, and then for the AT product to automatically disable both input and output translation to the scope of that control would go a long way in relieving this burden from our users. Mapping custom speech to Braille characters in such circumstances (which didn't require use of live regions) would also be helpful.

[sclower]

James,

I’d be happy to take the Apple-specific discussion offline if you or your team would be interested in more specifics and demonstrations about what I wrote above.

Best regards, Steve

[cookiecrook]

Please do. DM same username on Twitter or A11y slackers. Or email to first initial + last name at company.

[cookiecrook]

@jnurthen If we can have the Dec 17th deep dive time slot, I'll have two other Apple engineers joining the discussion for untranslated unicode braille input.

[jnurthen]

@cookiecrook sounds good. Look for confirmation in tomorrow's meeting minutes.

[cookiecrook]

Attempting to capture the outcome of the discussion:

1. New feature (maybe aria-brailleinput) that would have token values undefined and raw (or maybe true) . Token list may need to remain open-ended to allow future expansion via a TBD standard list of braille tables. (No such standard list exists today. See list item 3 below.)

The expectation from @samdooley (Pearson) is that these fields would allow both braille and other unicode characters, leaving disambiguation of the braille entirely to the web app, and allowing the web app to switch from translated to untranslated braille input by modifying the attribute in real-time. (e.g. while aria-brailleinput is true, braille comes in as raw unicode braille dot patterns. If the web app removed the attribute while the user typed, the braille input would switch back to the user's specified braille table, such as UEB, and non-braille characters would flow to the field.)

Prior to the call, I had not seen this desire to mix braille and non-braille in the same field... If necessary, it may limit the value of specifying the braille table on the element, because braille characters inside the field could be a mix and match of multiple tables, in the same way that a user can type multiple spoken languages in substrings of a single field. "Hello! Hola!"

Prior to hearing @samdooley's desire to mix braille and non-braille in the same field, @sclower (Desmos) and I discussed the possibility of utilizing the HTML <input pattern> attribute to limit the field's value to braille and whitespace characters. (If I did the RegEx correctly, that would be: <input pattern="[\s\u2800-\u28FF]*">) Note: the pattern attribute is currently limited to validation, and does not limit input. The goal of using pattern would be to avoid this braille input specifier being accidentally or erroneously applied. In other words, if a sighted user couldn't type anything in the field, the bug would be obvious and the bogus attribute would be removed quickly. If a bogus application only affected braille users, the bug would be much more likely to persist.

2. new feature (maybe aria-speechvalue) that could be used as a static "spoken" value for braille text, and potentially act like a speech-only live announcement during manipulation of the field.

A few people (@samdooley, @sclower, and @sinabahram IIRC) expressed a desire to use this similar to a live region announcement that would not be displayed in braille. For example, if the field contained "⠭⠐⠶⠽" (meaning "x=y" in UEB), the web app would need to track selection, and update the attribute accordingly. If the user selected or traversed across the "⠭" character, the web app should change the speech attribute value to "x"... Screen reader could then send "x" to the speech engine, but leave "⠭⠐⠶⠽" undisturbed on the display, denoted the cursor on the first cell.

I noted some concerns, namely:

• I am skeptical that most web apps (besides a few specialties, like Pearson, ETS, and Desmos) would handle real time selection events and braille translation accurately and performantly.
• There are Mac versus Windows differences in what users expect to hear with caret movement, and this is further complicated by user preferences.
• There no clear indication of what should be spoken in the case of a single-cell selection of a two-cell character. In UEB, the equals sign (=) is represented by two cells "⠐⠶" so if the user's caret should cross or select "⠐", what should the value of the speech attribute be? "Dot 5"? If so, the traversing selection across the cell would speak "x, dot 5, dot 2356, y" instead of the expected "x=y"
• If something like aria-speechvalue were used outside of the braille input context, it could be a candidate for abuse, as it may allow malicious developers to mask input from the user

3. There could be a separate effort to standardize an international list of braille table identifiers, similar to international country and language codes... It might be possible to leverage the lang attribute for braille. Coordinate with i18n groups at W3C. This would not need to block the braille input discussion.

4. Assuming the list of braille table identifiers is specified, it may be useful to use this outside the context of input. For example: <span lang="brl/en/ueb">⠭⠐⠶⠽</span> or similar.

[jnurthen]

@pkra please move to a later release if this is too big for 1.3

[pkra]

@jnurthen done.