Support for entry of contracted braille via a braille keyboard

[nvaccessAuto]

Reported by jteh on 2012-06-12 06:11 (Spun off #808.)

Supporting input of contracted braille is much more complicated than computer braille, as we need to determine when to translate a chunk of input, the user experience before translation of a chunk occurs, etc. Blocked by #808

[nvaccessAuto]

Comment 2 by ragb on 2012-07-08 14:47 I believe contracted braille input is really realy hard to do, specially expecting a good user experience. I haven't seen any good support for portuguese contracted braille input anywhere. I don't know english braille in detail but I think we may have similar problems such as:

• The same dot pattern (or multiple dot cells pattern) meaning completly different things. I.G. in portuguese, the pattern 126-3 might mean opening parentises or the characters "ê.". In portuguese the "o" character (135) is the same as the greater-then character.... With contractations the mess is even greater. So... a good experience, seems something very relative :).
• Special signs (such as number-sign, caps sign, etc.) also mean stuff such as "#",, "_"...
• Noone standardized and adapted braille to the needs and cenarious that came with tecnology and computers in particular. So many times there are no rules to follow, neither we don't know what to code neither users know what to expect many times. On braille input either then 8 dots, everone events their own rules (compare braille input in osx, in Jaws, or in brltty)....

Anyway, this is realy something that I'd like to have in NVDA (or anywhere else), but is something not trivial, in my opinion.

[nvaccessAuto]

Comment 3 by Ahiiron on 2012-07-12 17:35 Just voicing my support for this. It would be a very painful/complex thing to do right, but man, it would rule. Then I can narrow down my display I want to purchase to ones equipped with a Braille keyboard.

[nvaccessAuto]

Comment 4 by MHameed on 2012-07-13 13:23 I am slowly working on getting liblouis shaped up in terms of testability, correct tables etc. backtranslation is working but has several problems, but I am hoping that issues will be fixed so that nvda and orca can use the backtranslation features to provide the requested support in the Medium term.

[nvaccessAuto]

Comment 5 by nvdakor on 2012-07-13 16:36 Hi, In that case, I think we should work on a new branch, as this is something that'll require some extensive testing by end users.

[nvaccessAuto]

Comment 6 by ragb on 2012-12-04 00:55 Hi,

When implementing #808 I tried to use a literary braille table for back-translation. Of course the results were poor, since the goal for that ticket was computer braille input. In the meantime, I came with the following simple idea that might be a good starting point for this. (this requires familiarity with the work beeing developed on #808 and the !brailleInput branch).

Instead of translating a character as soon as a braille keyboard command is entered, we can back-translated the braille dots only when a full word is entered, that is when a space bar is pressed (this can be argued). Moreover, if no space bar is pressed after a while, we can assume that the input is done and the back-translation should be sent to the operating system anyway (this requires a timer). This way we can translate full words (or more characters at a time) instead of just one. The test being entered is not spoken or shown in braille until is sent, but when typing braille one can't read it at the same time anyway.

Steps for integrating this in #808 would be something like:

1. In brailleInput.BrailleInputHandler} create a buffer and a timer. When some braille pattern either tan space is entered add it to the buffer and reset the timer.
2. If space is entered, flush the buffer (back-translate and send to Windows, including the space character), and stop the timer (we don't need it while nothing is on the buffer).
3. If no space is entered but the timer comes to zero, flush the buffer anyway.

I implemented a scatch prototype (I need to merge it with new !brailleInput changes) and it seems to work reasonably with english grade 1 braille. With portuguese is a nightmare :(. I'm using 1 second for the described timer, but I believe this value most be configurable.

I supose this is simliar to what VoiceOver on the Mac is doing, but I have no way to confirm it. However, I believe we can do much better and refine this simple algorithm, specially if liblouis evolves in back-translation.

[nvaccessAuto]

Comment 7 by jteh on 2012-12-04 01:18 This is a good proof of concept. However, the timeout approach has been very harshly criticised by iOS users because a user doesn't think of the timeout and may take longer. A configurable timeout helps to mitigate this to some extent, but it's still not ideal. Also, not being able to read what has been entered could be bad for long words. These are the difficult questions i don't have answers for, but they need to be considered.

[nvaccessAuto]

Comment 8 by jteh on 2012-12-04 01:20 This might be obvious, but please keep any work on this in a separate branch from brailleInput. Thanks!

[nvaccessAuto]

Comment 9 by ragb (in reply to comment 7) on 2012-12-04 12:58 Replying to jteh:

This is a good proof of concept. However, the timeout approach has been very harshly criticised by iOS users because a user doesn't think of the timeout and may take longer. A configurable timeout helps to mitigate this to some extent, but it's still not ideal.

I think there should be a way to stop the timeout, that is, only consider the word entered when the user presses space or enter. We could have too different options: either provide a command to stop the timer (some chord, per haps), or make possible to configure the timout to infinity (no timeout :)).

However that rings another problem. If the user is writing a word and types something by mistake, he should be able to delete characters (using dot7, maybe), and type the right patterns to finish the word. This obviously makes the need for showing the user what's being entered...

Also, not being able to read what has been entered could be bad for long words.

Yes. I can't come with a good solution to this. Do you think having a custom braille region showing when the user is entering a word is possible? That region would contain the input being entered (in braille) and is to be updated whenever the user enters or deletes a braille character. I don't know if this brings a good user experience or not, but that is what I can think of. It's somehow resembles input composition.

These are the difficult questions i don't have answers for, but they need to be considered.

Me neither. However, I believe with some brainstorming we can come with better solutions from most of what is available (i.g. IOS).

[nvaccessAuto]

Comment 10 by nvdakor on 2012-12-04 15:06 Hi, I think we might want to also consider how notetakers handle braille input as well:

• From what I've seen with notetakers, they do instant translation on the fly based on the current contracted braille code in use. For example, if I want to type the word "testing" on my Apex (at least using its wordprocessor), I would type, "t, e, st sign". Then KeySoft reads, "t, e, st sign". Then I use read current word command and KeySoft reads "test". Then I type "ing sign", and KeySoft reads "ing". Then if I invoke read current word, KeySoft reads "testing".
• The downside of iOS is that it doesn't handle contractions which are conscenants - that is, a word that ends in a conscenant which is a word when contraction rule is applied. For example, if I type the word "book", I might type "boo". Then I would pause for about a second, then type "k", but VoiceOver would insert the word "knowledge".
• A problem with time-out approach would be that there might be a delay when a key is pressed and when the char is inserted (especially for Bluetooth displays and if the user's display is far from the computer). Also, this poses a problem for a number of foreign languages (in case if we wish to extend this concept to foreign contracted braille codes, which is something that international users would want to see in a future release". For this reason, I think we need to work with John, Mesar and other LibLouis people on perhaps implementing "smart back translation" based on incoming dots and whether a space surrounds such a dot. In order for this to work, we may want to use a braille rule (at least for English contracted braille codes) where many contractions comes after certain dots such as 4-6, 5 and so forth. For example, the algorithm could go something like:
• Find out which dot was pressed on the braille keyboard.
• If the dot corresponds to any one of the special contraction prefix, then display the actual dot on the braille display while withhelding the dot from the screen e.g. store the dot on a variable.
• If the user presses dot 7, delete the dot from the braille display and set the var's value to "none" (in C++, it'll be "").
• After the dot has been pressed, if the user types a letter or a sign corresponding to the contraction, then ask LibLouis to backtranslate the prefix dot and the contraction letter and send it to the screen. In doing so, some rules may need to be specified:
• Take care of punctuation: in this case, we can check for presence of special characters such as space, line feeds, etc.
• If a number sign has been entered, initially treat it as the contraction corresponding to the number sign "ble in U.S. grade 2". To enter number, one would type the number sign again, or type dots 3-4-5-6 with dot 7. Potential flaws:
• What if a punctuation is inserted in the middle of numbers or letters?
• In case we will be assuming that the keyboard has 8 buttonns; what if there are keyboards with only six? I think, for short term, asking the user to type dot 7 with dot patterns for punctuation or providing one or two extra commands for treating the next dot press as special char would work. Then based on the characters which surrounds it, the entered punctuation or symbol can be back translated correctly (mostly for brackets and quotation marks). Another approach would be to utilize the dot decision algorithm such that certain dot combinations (mostly dots 5-6) can be used to decide whether the next entered character is a letter or something else (or using dot 4 would work, but then we may get issues with accents). Just my thoughts.

[nvaccessAuto]

Comment 11 by jteh (in reply to comment 10) on 2012-12-07 17:04 Replying to nvdakor:

• From what I've seen with notetakers, they do instant translation on the fly based on the current contracted braille code in use. For example, if I want to type the word "testing" on my Apex (at least using its wordprocessor), I would type, "t, e, st sign". Then KeySoft reads, "t, e, st sign".

It's significantly simpler for notetakers to do this because the entire platform is custom designed. For a screen reader, there are three problems:

1. We can't send the characters to the application as they're being entered, as any new Braille cell could change the translation completely. For example, if the user is typing "tgr" for "together", we can't send "tg" because it needs to be changed once the "r" and space is received.
2. If we don't send characters to the application, they won't get spoken as they're being typed. (Can anyone tell me how this works on iOS? I imagine it doesn't speak what's been typed until the timeout.)
3. If we don't send characters to the application, they don't appear on the screen or, more importantly, the Braille display. NVDA is designed to display text from applications (which is normally exactly what the user wants). We could display the entered dots separately, but it would be extremely difficult to integrate the dots being entered with the rest of the actual text. As Rui points out, this is more like input composition, which is normally displayed separately from the actual text. This is why iOS and other screen readers implement a timeout. They want to be able to send the text to the application (and thus show the text to the user) without waiting for the end of each word.

One other problem for NVDA is that liblouis can't tell us what individual special Braille cells mean. For example, it can't tell us that dots 3-4 is the "st" sign unless it's part of a full translation.

• The downside of iOS is that it doesn't handle contractions which are conscenants - that is, a word that ends in a conscenant which is a word when contraction rule is applied. For example, if I type the word "book", I might type "boo". Then I would pause for about a second, then type "k", but VoiceOver would insert the word "knowledge".

That's because of the timeout.

• A problem with time-out approach would be that ... this poses a problem for a number of foreign languages

Out of interest, why is a timeout more of a problem for some languages than others?

Your ideas about special dot decisions in liblouis would require some pretty significant changes both to liblouis itself and all of its tables. I'm not necessarily saying it's a bad idea, but it'd probably be a very long way off.

[dkager]

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days. I know the Dutch (nl-NL and nl-BE) tables aren't written for backtranslation and probably fail for the parts that use multi-pass rules. (Side question: why does NVDA disable the multi-pass facilities when forward translating?) For UEB there is a lot of great work done by the APH that is pending review and possible inclusion in a future version of liblouis. This code too appears to deal only with forward translation. I don't know how much of the backtranslation functionality is covered by unit tests but this might be a good place to start.

[jcsteh]

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days.

liblouis is all we have. I take your point, though. I guess if it's really that bad, we'll have to push for improvements to the tables and/or liblouis before we can release this.

(Side question: why does NVDA disable the multi-pass facilities when forward translating?)

Because multi-pass breaks the input/output position mapping. I have had grand ideas of completely rewriting that position mapping code for a while, but I've never gotten to it. Arend Arends contributed a patch for this, but it hasn't been integrated yet and I believe there are some outstanding concerns.

[dkager]

The next release of liblouis is planned for next Monday and should include the patches contributed by Arend, Bue, etc.

I’d be happy to look into backtranslation for the Dutch tables, though if it requires extensive code changes it’ll have to wait till the new year.

From: James Teh [mailto:notifications@github.com] Sent: Thursday, November 26, 2015 02:16 To: nvaccess/nvda nvda@noreply.github.com Cc: Davy Kager mail@davykager.nl Subject: Re: [nvda] Support for entry of contracted braille via a braille keyboard (#2439)

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days.

liblouis is all we have. I take your point, though. I guess if it's really that bad, we'll have to push for improvements to the tables and/or liblouis before we can release this.

(Side question: why does NVDA disable the multi-pass facilities when forward translating?)

Because multi-pass breaks the input/output position mapping. I have had grand ideas of completely rewriting that position mapping code for a while, but I've never gotten to it. Arend Arends contributed http://www.freelists.org/post/liblouis-liblouisxml/Update-output-positions-during-multi-pass-forward-translation-only a patch for this https://github.com/liblouis/liblouis/pull/133 , but it hasn't been integrated yet and I believe there are some outstanding concerns.

— Reply to this email directly or view it on GitHub https://github.com/nvaccess/nvda/issues/2439#issuecomment-159772600 . https://github.com/notifications/beacon/AC9y8Y5Tk75wiMamPMzj_9dVTbbCMaXXks5pJlTMgaJpZM4GpUFG.gif

[derekriemer]

1. select UEB
2. type dots a 3456-1
3. type space. a is actually typed rather than 1

[derekriemer]

In UEB, type hello! 125-15-123-123-135-235 Result: ff is typed rather than !

[derekriemer]

Type 542 press space delete until 2 type b 12 a 2 should appear, rather but appears.

[derekriemer]

As a user, I'd find it useful to force NVDA to translate with out forcing an enter or space.

[derekriemer]

Great work, that's all my observations as of now.

[jcsteh]

@derekriemer commented on 12 Oct. 2016, 3:23 am AEST:

1. select UEB
2. type dots a 3456-1
3. type space. a is actually typed rather than 1

I can't reproduce this. I get "1" as expected.

@derekriemer commented on 12 Oct. 2016, 3:27 am AEST:

In UEB, type hello! 125-15-123-123-135-235 Result: ff is typed rather than !

I can't reproduce this either. I get "!", not "ff", once translation is complete. However, if you use speak typed characters, you will hear "ff" when you type the dot 2-3-5 because we can't actually be certain of what the character will be until translation is complete. Once you press space, the text actually entered will be "!", though.

@derekriemer commented on 12 Oct. 2016, 3:28 am AEST:

Type 542 press space delete until 2 type b 12 a 2 should appear, rather but appears.

Unfortunately, we can't fix this. Translation is based only on the word you're typing, not text at the cursor. Doing otherwise would require that the input and output tables always be the same, which is not always desirable. Also, later cells can change the text produced by earlier cells, so if that text has already been entered, that's problematic. Perhaps this needs to be documented more clearly, though.

@derekriemer commented on 12 Oct. 2016, 3:31 am AEST:

As a user, I'd find it useful to force NVDA to translate with out forcing an enter or space.

I didn't add this because I felt it might be confusing for users. You should be aware that if you pressed this, it would reset translation. For example, typing "b", then ending translation, then typing "b" again would result in "butbut", not "bb". Is that what you expect?

[derekriemer]

@jcsteh commented on Oct 11, 2016, 4:57 PM MDT:

@derekriemer commented on 12 Oct. 2016, 3:23 am AEST:

1. select UEB
2. type dots a 3456-1
3. type space. a is actually typed rather than 1

I can't reproduce this. I get "1" as expected.

I can't any more. I think what I meant to say is things like that found in a password would do fill1 would go as fillable This can be fixed by using the new press a key to force translation, which is actually nice for this case. @derekriemer commented on 12 Oct. 2016, 3:27 am AEST:

In UEB, type hello!

125-15-123-123-135-235

Result: ff is typed rather than !

I can't reproduce this either. I get "!", not "ff", once translation is complete. However, if you use speak typed characters, you will hear "ff" when you type the dot 2-3-5 because we can't actually be certain of what the character will be until translation is complete. Once you press space, the text actually entered will be "!", though.

Not here. @josephsl can reproduce this as well. I also can type "hello!!!!!" and I get "hello! to to to to" in multiple braille tables.

@derekriemer commented on 12 Oct. 2016, 3:28 am AEST:

Type 542

press space

delete until 2

type b 12

a 2 should appear, rather but appears.

Unfortunately, we can't fix this. Translation is based only on the word you're typing, not text at the cursor. Doing otherwise would require that the input and output tables always be the same, which is not always desirable. Also, later cells can change the text produced by earlier cells, so if that text has already been entered, that's problematic. Perhaps this needs to be documented more clearly, though.

Could it be possible to make NVDA work in this case only if input == output?

@derekriemer commented on 12 Oct. 2016, 3:31 am AEST:

As a user, I'd find it useful to force NVDA to translate with out forcing an enter or space.

I didn't add this because I felt it might be confusing for users. You should be aware that if you pressed this, it would reset translation. For example, typing "b", then ending translation, then typing "b" again would result in "butbut", not "bb". Is that what you expect?

The benefit of this is when entering something like brailleit4u or something with a number in a word.

[jcsteh]

a is actually typed rather than 1

My sincere apologies. I can in fact reproduce this now that I'm running liblouis 3. Previously, I was running an outdated liblouis. I never considered the possibility that they might have regressed it in liblouis 3. :(

Both this and the "hello!" case are due to bugs in the liblouis 3 UEB tables. According to this mailing list post, UEB backtranslation issues are being addressed at present. That post also requests people to report any backtranslation issues they encounter.

Unfortunately, we can't fix this. Translation is based only on the word you're typing, not text at the cursor. Doing otherwise would require that the input and output tables always be the same, which is not always desirable.

Could it be possible to make NVDA work in this case only if input == output?

No. Later cells can still change the text of earlier cells. So, if the previous word was "tg" and you type "r", it has to try to erase the "g" and insert "ogether". Things get even more complicated if the user drops the cursor into the middle of a word. And those examples are without even trying to come up with obscure edge cases.

As a user, I'd find it useful to force NVDA to translate with out forcing an enter or space.

The benefit of this is when entering something like brailleit4u or something with a number in a word.

Well, you can do that without ending translation; UEB does support that (table bugs notwithstanding). Regardless, I've implemented this now.

[LeonarddeR]

@jcsteh: Reported this earlier on IRC, but for refference. When you select UEB1 as input table and enter dot 6, dot6 will stick on the braille display until you do another braille input gesture. This is expected of course, however it leads to slightly unexpected behaviour when you press dot 6 and than change your mind and start typing on the pc keyboard. Dot 6 will stay until the braille buffer until you switch windows or resume braille input.

[jcsteh]

Deferring to 2017.1 due to outstanding issues. See https://github.com/nvaccess/nvda/pull/6449#issuecomment-258073729 for details.

[feerrenrut]

Consider fixing "Make Unified English Braille the default Braille code" #6952 as part of this issue.

[josephsl]

Hi, I propose putting that all under Liblouis 3.1.0 PR (#6935). Thanks.

From: Reef Turner [mailto:notifications@github.com] Sent: Thursday, March 9, 2017 10:07 PM To: nvaccess/nvda nvda@noreply.github.com Cc: Joseph Lee joseph.lee22590@gmail.com; Mention mention@noreply.github.com Subject: Re: [nvaccess/nvda] Support for entry of contracted braille via a braille keyboard (#2439)

Consider fixing "Make Unified English Braille the default Braille code" #6952 https://github.com/nvaccess/nvda/issues/6952 as part of this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/2439#issuecomment-285586430 , or mute the thread https://github.com/notifications/unsubscribe-auth/AHgLkBfcJhQTN6n0Qr9Z9xEskWeV4dU1ks5rkOiEgaJpZM4GpUFG .

[josephsl]

Hi, er, I think @JcSTeh may have closed this but not referenced this issue directly. As far as implementation goes, it is part of NVDA since 2017.3. Thanks.

From: Adriani90 notifications@github.com Sent: Thursday, December 6, 2018 1:20 PM To: nvaccess/nvda nvda@noreply.github.com Cc: Joseph Lee joseph.lee22590@gmail.com; Mention mention@noreply.github.com Subject: Re: [nvaccess/nvda] Support for entry of contracted braille via a braille keyboard (#2439)

I see that the discussion here is very comprehensive. Is this issue now solved? Or could someone give a short update on this? Thank you.

@josephsl https://github.com/josephsl sephsl, @jcsteh https://github.com/jcsteh

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/2439#issuecomment-445033290 , or mute the thread https://github.com/notifications/unsubscribe-auth/AHgLkJgfc_AbztaBwEqPl3cnPzssV7pzks5u2YoDgaJpZM4GpUFG .

[josephsl]

Hi,

Never mind about my last comment. Perhaps Adriani didn't see the issue was actually closed.

Thanks.