Let screen readers catch cursor position

shindere commented 2 years ago

TErmux is almost usable with a screen reader like brltt.

One major barrier to its use, though, is that brltty does not know where the cursor is. More specifically, no matter her the crr iviually, brlttyshows it at the botom-right of the reen.

Could this please be fixed?

If made usable with brltty, termux could become a fundamental app for visually impaired persons, especially since braille notetakesrs like HIMS [Braille Sense 6]() are running Android.

agnostic-apollo commented 2 years ago

Hey. The issue is that the TerminalView is not really a text field (EditText) used by android accessibility service and so there is no concept of "cursor position" for it. Consider it more like an image with pixels. Termux app does publish the text shown in the terminal as contentDescription added in #344.

My guess was that cursor being reported at bottom right was because of terminal not being a text field, which I confirmed by installing the brltty android app from https://play.google.com/store/apps/details?id=org.a11y.brltty.android as documented in https://brltty.app/archive/Android/brltty-on-android.html. I enabled logcat logging in accessibility settings and enabled Cursor Tracking log category. Brltty logged the following entries when I opened the terminal with extra keys mode, then switched to text input view mode, then typed 3 characters and then switched back to extra keys mode. The text input view is an EditText, so it reports the cursor position correctly. The cursor position for terminal is always 0,1, but changes to 0,40 in input text view mode and updates when text is changed. There are some extra entries generated during swipe action when switching to text input mode.

The log entries are from brltty/Programs/update.c

W/brltty: cursor tracking: scr=237 csr=[0,1]->[0,47] win=[0,1]->[0,47]
W/brltty: cursor tracking: scr=237 csr=[0,47]->[0,42] win=[0,43]->[0,43]
W/brltty: cursor tracking: scr=237 csr=[0,42]->[0,40] win=[0,41]->[0,41]
W/brltty: cursor tracking: scr=237 csr=[0,47]->[0,40] win=[0,41]->[0,40]
W/brltty: cursor tracking: scr=237 csr=[0,40]->[1,40] win=[0,40]->[1,40]
W/brltty: cursor tracking: scr=237 csr=[1,40]->[2,40] win=[1,40]->[2,40]
W/brltty: cursor tracking: scr=237 csr=[2,40]->[3,40] win=[2,40]->[3,40]
W/brltty: cursor tracking: scr=237 csr=[3,40]->[0,1] win=[3,40]->[0,1]

As for whether this can be fixed or not, I am currently not sure since I don't really have experience with accessibility services. However, there seems to be support in android for custom views as documented at https://developer.android.com/guide/topics/ui/accessibility/custom-views. Maybe custom events can be published with updated AccessibilityNodeInfo.EXTRA_DATA_TEXT_CHARACTER_LOCATION_KEY in View.onInitializeAccessibilityNodeInfo(). Chromium seems to be doing something similar in OWebContentsAccessibility. But not sure if its the right thing for termux terminal.

I will try looking into this. I don't have a braille screen reader device, so that may be an issue too and may need someone who does for testing.

shindere commented 2 years ago

Many thanks for your thorough investigation!

I do have a braille display and will be happy to test and assist in any possible way!

@sthibaul may have an opinion about the way to go, although I am not sure how familiar he is about accessibility on Android.

agnostic-apollo commented 2 years ago

You are welcome. Cool, will let you know in case I need testing.

shindere commented 2 years ago

Thanks. Really. For blind people, fixing this issue would be a huge enhancement.

agnostic-apollo commented 2 years ago

Yeah, I understand and hope this gets fixed somehow. I'll try prioritizing this.

shindere commented 2 years ago

agnostic-apollo (2022/02/05 13:03 -0800):

Yeah, I understand and hope this gets fixed somehow. I'll try prioritizing this.

It's already wonderful you got interested in this and went that far. YOur comment was awesome with all the details and links you provided.

sthibaul commented 2 years ago

I don't know anything about the accessibility stack on Android.

webczat commented 2 years ago

note there is an alternative accessibility method which is using an actual linux specific console screenreader. had some success with fenrir but it required proot setup and some way for it to contact the external pulseaudio, I don't even remember what I did.

shindere commented 2 years ago

The issue reported here is independent of the screen reader at use, I think.

For a screen reader to work, it must have a way to "see" what is displayed on the screen.

When I last tested, it was almost the case with Termux. The only problem I could see during my rough tests was that the reported position for the cursor was wrong: it was constantly reported to be at the bottom-right of the screen, even in Emacs, for instance.

Meanwhile, I got my Android braille display so I am available to perform tests.

agnostic-apollo commented 2 years ago

I was actually looking into this yesterday before you guys messaged.

I have managed to find a way to send terminal cursor position via [AccessibilityNodeInfo.extras](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#getExtras()). There doesn't seem to be any other valid fields that could be used. Moreover, more custom data can be sent as well or specially returned if requested by accessibility apps.

Check #2840. You can download termux apk build from https://github.com/termux/termux-app/actions/runs/2509869570. Note that if you installed termux from F-Droid, then installation github build will fail since it will have different signing keys, so you would need to uninstall termux and all plugin apps first. Check https://github.com/termux/termux-app#Installation.

You would also need to apply the brltty app patch to get the values. Currently, its just logging the values and not setting screen reader cursor. Log Accessibility Events will have to be enabled in advanced brltty settings. Let me know if I should open a pull request for this.

The issue is that I don't have experience with screen readers or terminal support for screen readers and don't know how this should ideally be implemented and how useful cursor position is. The terminal is two dimensional, but screen readers seem to be one dimensional, basically a single row with limited text characters. So there is some mapping going on as well. Moreover, in non text edit mode (outside nano, etc), the user may want to read text at the top of the terminal and cursor would be at the bottom waiting for new commands, so content description would have to start at the top and not at the cursor at command line. In text edit mode, the cursor would be aligned with text to be read, so likely more ideal. But even in text edit mode, when the cursor is at extreme right of terminal, there may still be text available to the right on the line of the file being read, in that case cursor position would still be same, even after horizontal scrolling.

The brltty app is also not considering if text field is [multiline](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#isMultiLine()) and cursor tracking trky is at wrong offset and even returning to text field after app switch seems to be broken.

Are there some standards available that should be supported by terminals for screen readers? There is some info on cursors at http://brltty.app/doc/Manual-BRLTTY/English/BRLTTY-5.html#ss5.1

I need more info on what's expected by a screen reader or a blind person so that an ideal system can be designed since obviously there are issues.

Console screen readers are available too and it might be possible to add native packages for them in termux. There was also work being done audio desktop like emacspeak at https://github.com/termux/termux-packages/issues/6504

Android accessibility also has support for virtual views. So, for the terminal, I should be able to implement something like 3 modes. Possibly extras keys can be used to switch between the modes, but users would have to configure the extra keys, so not sure how convenient that would be at app install.

Window mode in which entire terminal text is set as content description and terminal is considered a single view.
Line mode in which each row is a separate view, allowing users to move between rows and having their text read.
Character mode in which each row and column is a separate view.

Currently, termux is using window mode and setting content description to the entire terminal text that is visible so TalkBalk starts reading text from the top everytime, which would obviously be annoying and time wasting. So line and character mode may be useful as well. It would also have to be decided if view mode should apply to all opened terminal sessions or only current one. I also don't know how screen readers handle incoming text, I am assuming its buffered and can be scrolled through and read separate from android.

Android has a sample for virtual views as well.

https://github.com/Miserlou/Android-SDK-Samples/blob/master/ApiDemos/src/com/example/android/apis/accessibility/AccessibilityNodeProviderActivity.java

Chromium also has extended accessibility support.

https://github.com/chromium/chromium/tree/main/content/public/android/java/src/org/chromium/content/browser/accessibility

Improving accessibility support via Android APIs would require collaboration will brltty devs, possibly @DaveMielke who seem to be handling android side. Any input on above would be appreciated.

webczat commented 2 years ago

unsure if screenreaders which use speech can easily be made to work well with any kind of terminal on android. I hope it's possible but might not be, as their usage is kinda different compared to desktop screenreaders. Might be far better in case of braille, but I don't have any braille display. Console screenreaders can work in theory, in practice I managed to get a setup using proot archlinux and pulseaudio, it worked but was noticeably slower to be honest. Console screenreaders which are not linux specific often grab a pseudoterminal and don't care about anything else, just intercepting characters coming into or out of a pseudoterminal, so no support from termux is needed in that case except things these screenreaders want to use which normally don't work or things like sound/speech synthesis.

agnostic-apollo commented 2 years ago

Well, proot is going to be slow, hence native support if possible would be better.

webczat commented 2 years ago

well if any console screenreader would work without proot that would be something too. the one I have in mind probably doesn't.

DaveMielke commented 2 years ago

Hi:

Since you're including me in this discussion, I'm assuming that you'd like me to contribute to it. Could you please explain to me what your goal is? That'd help me know which kinds of answers you'd like.

You mention brltty. It's already a fully functional screen reader so what are you trying to offer on top of it?

You mentioned that brltty doesn't work with multi-line text views. That isn't correct. It works fine with them.

shindere commented 2 years ago

Dave,

I am the person who initially opened the issue. Are you able to read its content?

In a few words: termux is an Android app which lets you use command-line tools on an Android device. You can install packages, edit files, etc.

When you isntall both termux and brltty on an Android device, brltty already has access to the screen content exposed by Termux. The last time I tested, the only thing brltty couldn't get was the position of the cursor.

How to fix this is what is being discussed here, but it may be helpful if you were able to read the issue from the beginning.

If you are not able to do so in a comfortable way, just let me know and I will send you a text export of everything that has been posted to it.

DaveMielke commented 2 years ago

[quoted lines by Sébastien Hinderer on 2022/06/16 at 13:13 -0700]

In a few words: termux is an Android app which lets you use command-line tools on an Android device. You can install packages, edit files, etc.

So all I need to do is to install the termux app?

When you isntall both termux and brltty on an Android device, brltty already has access to the screen content exposed by Termux. The last time I tested, the only thing brltty couldn't get was the position of the cursor.

That should hopefully be easily fixable. Is termmux using a TextView for the console content or is it using some other kind of view.

What I can say for sure is that if a TextView is being used then brltty has no problem at all knowing where the cursor is. Perhaps, therefore, I should ask a different question. How does termmux not knowing where the cursor is manifest itself?

How to fix this is what is being discussed here,

Of course I'll get around to actually trying it, but I'd still appreciate it if someone could describe for me how termux is rendering the console and also how it's asking where the cursor is.

but it may be helpful if you were able to read the issue from the beginning.

I much prefer to tackle any issue from scratch.

What I'm hoping is that termux is using a standard TextView and that it's simply asking which text is currently selected to know where the cursor is. No text being selected is indicated by the start and the end of the selection being both set to the same offset, which is exactly where the cursor is. Put another way, text not selected is simply the devolved case of the selected text having a length of zero.

agnostic-apollo commented 2 years ago

Could you please explain to me what your goal is?

The short term goal is to fix cursor position of the terminal being assumed by brltty.

The long term goal is to improve accessibility support for termux app and terminal. I don't know what kind of data or features screen readers or blind people would want from a terminal in terms of easier accessibility, so those goals are yet to be decided on. I also don't have a screen reader myself to test stuff out.

You mentioned that brltty doesn't work with multi-line text views. That isn't correct. It works fine with them.

The vertical offset doesn't seem to be correct in logs and is not 0 on first line and changes and something like 13 as per my tests, also gets messed up if I switch back to gmail email send edit screen, I don't know if it works well with the screen reader even with faulty values, or at least what I am assuming are faulty.

So all I need to do is to install the termux app?

You can install termux from https://github.com/termux/termux-app/actions/runs/2509869570 if you want to test the pull request #2840. You need apt-android-7 variants, like termux-app_v0.118.0+bd6347d-apt-android-7-github-debug_universal, unless you know your phone's arch. Don't use apt-android-5 variants, they are only for android 5/6, with legacy/unsupported packages.

That should hopefully be easily fixable. Is termmux using a TextView for the console content or is it using some other kind of view.

Termux uses a custom TerminalView, which is not a TextView or EditText.

https://github.com/termux/termux-app/blob/master/terminal-view/src/main/java/com/termux/view/TerminalView.java

How does termmux not knowing where the cursor is manifest itself?

Termux knows the cursor position. But since its not an EditText, brltty and other accessibility apps don't know the cursor position. Terminal apps/packages/text editors get position with a request for Cursor Position Report (CPR), but android accessibility can't use that obviously.

No text being selected is indicated by the start and the end of the selection being both set to the same offset, which is exactly where the cursor is. Put another way, text not selected is simply the devolved case of the selected text having a length of zero.

Issue with EditText is that its linear. The start and end index is based on position inside the linear text. The terminal is 2D, so cursor position is the row and column. However, we can extract all the text out of the terminal and convert row and column into a start and end index inside the text, but I am not sure if users want that, since users may want actual row and column when using a terminal.

I have opened the pull request #2840 in which termux sends the row and column via AccessibilityNodeInfo.extras to accessibility apps like brltty. You can apply this patch in your app to get the values. I don't know how you will get the row and column to the screen reader though, since they are not start and end index as already explained.

DaveMielke commented 2 years ago

I may have expressed myself the wrong way around. The real issue is how termux is showing where the cursor is. The right way to do that within a TextView is to set both the selection start and selection end to the same offset - that of the cursor's position.

I also feel the need to raise a cautionary warning: Please don't try to outsmart screen reader apps. Let them do what they do, and help, rather than try to override, them.

All this talk about doing things like setting the content description is unnecessary. If there's a perceived need to do that then that's guaranteed to be the wrong approach. And, yes, if you're trying to use the content description to render the console's text then the whole concept of where the cursor is has been lost. The content description is only ever meant to be a helpful textual description as to what a widget contains.

Let's say you have a graphic of an analog clock face. Then the content description should be set to the text for the time because a screen reader app can't make sense of the graphic. That'd be a proper use of the content description property of a widget. Another would be when a button shows what it does via an image, in which case its content description should be set to a brief textual description of what action the button will perform.

I suppose it's possible that you render the console text with some fancy font, which'd require you to use what roughly would amount to images on a canvas. No screen reader could make any sense of that, of course, which might be why you're trying to then use the content description property. If this is correct, then I'd say that a much better approach would be to have a mode setting which can do it the graphic way but which can also be switched to a TexstView way of doing it.

agnostic-apollo commented 2 years ago

Firstly, I should add that I wasn't the one who implemented setting the entire terminal text to content description. It was before my time.

Secondly, yes I understand that content description should basically only be used to describe the content.

What I want to do is start fresh. What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor. Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView, each line as a separate TextView or each character of each row and columns as a TextView. I discussed that in https://github.com/termux/termux-app/issues/2543#issuecomment-1157768473. But I am not sure what's ideal since I don't have experience with screen readers.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

https://wiki.termux.com/wiki/Touch_Keyboard#Extra_Keys_Row

https://wiki.termux.com/wiki/Touch_Keyboard#Text_Input_View

https://wiki.termux.com/wiki/Terminal_Settings

Also I can't just convert the terminal view itself to TextView since terminals are very complicated and not just text. I can only make it appear as a TextView, possibly with virtual views. I haven't tested that yet though.

Maybe we can bring attention of someone who has worked on console screen reader support, they may be able to give some helpful info.

DaveMielke commented 2 years ago

[quoted lines by agnostic-apollo on 2022/06/16 at 14:22 -0700]

Brltty isn't assuing anything. It works with each widget as best it can.

The long term goal is to improve accessibility support for termux app and terminal. I don't know what kind of data or features screen readers or blind people would want from a terminal in terms of easier accessibility, so those goals are yet to be decided on.

My challenge to that is that it makes no sense to build on a faulty foundation. It'd be very difficult to convince any screen reader maintainer to want to support some kind of overly complex mechanism which is known to be merely temporary.

You should be using standard OS capabilities as much as possible. If you don't then you'll never be actually improving accessibility. The most would be possibly improving it for, maybe, just one screen reader at a time.

I also don't have a screen reader myself to test stuff out.

Sure you do. Every Android system comes with TalkBack. All you need to do is to enable it.

The vertical offset doesn't seem to be correct in logs and is not 0 on first line and changes and something like 13 as per my tests, also gets messed up if I switch back to gmail email send edit screen, I don't know if it works well with the screen reader even with faulty values, or at least what I am assuming are faulty.

All I can say, here, is that you're wrong. I depend on those logs for brltty support, and can absolutely assure you that the values they report are totally accurate. In fact, I'm even wondereing why you'd dare to conclude that they aren't accurate.

Termux uses a custom TerminalView, which is not a TextView or EditText.

Then I'd suggest that, as mentiond in my earlier message, you have a mode which does use an EditText.

Termux knows the cursor position. But since its not an EditText, brltty and other accessibility apps don't know the cursor position.

If you really want to improve accessibility then, really, you need to cater to the screen reader apps rather than expect ech of them, separately, to cater to you.

Terminal apps/packages/text editors get position with a request for Cursor Position Report (CPR), but android accessibility can't use that obviously.

I highly doubt that's true for all (or even most) such apps. In any event, anyone who supports a screen reader app is looking for compliance with OS standards, not externally imposed conventions.

Issue with EditText is that its linear. The start and end index is based on position inside the linear text. The terminal is 2D, so cursor position is the row and column. However, we can extract all the text out of the terminal and convert row and column into a start and end index inside the text, but I am not sure if users want that, since users may want actual row and column when using a terminal.

Not so. Sure, the content of an EditText is a single string, but that string contains embedded newlines. Brltty, for example, turns it into a multi-line view so that each line can be individually addressed, rendered, edited, etc.

I have opened the pull request #2840 in which termux sends the row and column via AccessibilityNodeInfo.extras to accessibility apps like brltty. You can apply this patch in your app to get the values. I don't know how you will get the row and column to the screen reader though, since they are not start and end index as already explained.

But that'd require each screen reader app to know how to handle your specific, non-standard widget. Why should each separate screen reader app need to have such specialized knowledge? That's an incredibly fragile interface which probably won't interest many (if any). iSmply designing ones own solution and then telling others that they must implement it is, quite frankly, a no go!

DaveMielke commented 2 years ago

[quoted lines by agnostic-apollo on 2022/06/16 at 14:43 -0700]

Firstly, I should add that I wasn't the one who implemented setting the entire terminal text to content description. It was before my time.

That was a very bad decision. :-(

Secondly, yes I understand that content description should basically only be used to describe the content.

I'm glad. :-)

What I want to do is start fresh. What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor.

Presenting the console as a single TextView containing all of the text (i.e. none of the per line or per character stuff) would be the way to go. Screen readers have their own ways to navigate text and there's no need to try to be smarter than they are. Speech ones know how best to do it when speech is being used, braille ones know how to do it best for those who use braille, etc.

Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

That makes no difference to a screen reader. Just show the text and where the cursor is.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView,

This is what you should do.

each line as a separate TextView or each character of each row and columns as a TextView.

Both of these would make life much more difficult for a screen reader app.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

That may make a nice user interface, but all any screen reader app will ever care about is what the TextView is actually rendering.

shindere commented 2 years ago

Hello,

I am goiong to respond although I am not familiar at all with Android developemnt. In case you don't manage to find out what you need, please let me know and I will ask around (I have skilled people among my colleagues, for instances).

agnostic-apollo (2022/06/16 14:43 -0700):

What I want to do is start fresh.

That would be really cool. Thanks a lot for yoru interest and the effort you seem ready to invest in all this. I hope it will be benefical to the whole app.

What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor.

As Dave explained, I think, the more you use the widgets already provided by the OS or SDK, the likeliest it is things will "just work".

Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

Yes, we (users of braille displays) are aware of that, but it is not really important. Indeed, when using a braille display, we can visit different fragments of what is on the screen, no matter where the cursor is. We can always locate the cursor, but we are not bound to it and can read even those parts of the screen where there is no cursor. This is all implemented by the screen reader and you, as the developer of an application, do not need to care about all this logic.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView, each line as a separate TextView or each character of each row and columns as a TextView. I discussed that in https://github.com/termux/termux-app/issues/2543#issuecomment-1157768473. But I am not sure what's ideal since I don't have experience with screen readers.

What maters to the screen reader is to be able to figure out thecontent of each line in terms of text and perhaps also the attributes (the color). The screen reader also needs to know where the cursor is at any time. I don't know how this has to be implemented in terms of Android widgets, so I can only provide rather high-level requiremnets.

To give you an example, to access the Linux text-console, screen readers typically use /dev/vcsa. That's to show you the kind of information which is needed.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

I think we will need to figure out together whether these functionalities would make sense and be useful for a blind user using termux from a braille display and a keyboard. My guess is that in such situation we would already be able to do HOME, END etc., without the need to interact with the buttons, but I may be wrong and there may be other functions which are not reachable from the keyboard.

In any case, my asusmption is that this tool bar is not really a priority and that it's best to start by focusing on the terminal itself, at least as a first step.

I hope this helps and am willing to provide as much support as required. I am also available for direct exchanges through an instant messaging system or for Audio/Video calls.

agnostic-apollo commented 2 years ago

I'll get back soon, got busy with other stuff. I implemented cursor positioning based on if both start and end selection indexes are same as suggested and as used by brltty. Need to do more testing to ensure everything is right.

Terminals are word wrapped as well, so standard selection index does not apply and each word wrap is a separate line.

shindere commented 2 years ago

No pressure, no hurry.

I just wanted to make sure you know there is interest in your wokr and will to support.

At the moment I din't understand what you menat about word wrapping and text selection but if I were you I woulnd't lose time explaining because I suspect everything will become clear when I am testing and have real questions.

DaveMielke commented 2 years ago

[quoted lines by agnostic-apollo on 2022/06/24 at 08:13 -0700]

I implemented cursor positioning based on if both start and end selection indexes are same as suggested and as used by brltty. Need to do more testing to ensure everything is right.

Note that brltty also supports selected text. So, if text is selected (which, on Android, also means that there's then no cursor) just set the start/end of the selection as needed.

Terminals are word wrapped as well, so standard selection index does not apply and each word wrap is a separate line.

That shouldn't matter. Just place newline (\n) separators at the right places within the TextView, and set the cursor position (or text selection) according to offsets within that same TextView.

agnostic-apollo commented 2 years ago

Note that brltty also supports selected text. So, if text is selected (which, on Android, also means that there's then no cursor) just set the start/end of the selection as needed.

I want to add text selection support but issue with terminal is that selection can be done anywhere in scroll back history, so if user is at bottom and selection is somewhere above in history, the selection text won't be visible or part of currently viewed text. Or only end cursor will be part of text. Although, could be handled by if both start and end indexes are visible, but user may want to select non visible and visible text as well, so gets complicated.

________________
|               |
|   Scroll back |
|  ^............|
| ........^     |
|_______________|                   
|               |
| Viewable text |
| on screen     |
|               |
|_______________|

That shouldn't matter. Just place newline (\n) separators at the right places within the TextView, and set the cursor position (or text selection) according to offsets within that same TextView.

Not using TextView currently, just setting visible text to AccessibilityNodeInfo.setText() and cursor to AccessibilityNodeInfo.setTextSelection(). The text passed is word wrapped as it looks when a user views a terminal. But since terminal stores cursor as row and column, and not index inside the word wrapped text (after newlines), need to translate row and column to a single index inside the text passed. It seemed to be working fine in brltty logs, just want to ensure there are no wrong offsets or calculations.

DaveMielke commented 2 years ago

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

shindere commented 2 years ago

Dave Mielke (2022/06/24 09:16 -0700):

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

DaveMielke commented 2 years ago

[quoted lines by Sébastien Hinderer on 2022/06/24 at 09:18 -0700]

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

Brltty sees the whole TextView, even if all of it doesn't fit on the physical screen. What happens to be visible on the screen isn't relevant to a braille user just like what can fit on a braille display isn't relevant to a sighted user. Another way to put it is that both scroll independently and differently from one another.

agnostic-apollo commented 2 years ago

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

In context of terminal, I don't think that would be wise. Firstly, user wouldn't know what's actually on the screen if I pass entire terminal text and Talkback will also start reading from top of scroll back history every time instead of visible terminal. Secondly, the terminal by default has 2000 rows and somewhere around 80 columns, so passing 2000*80=160KB on each terminal text update possibly happening many times a second is not really efficient and if accessibility is using binder transactions, which likely it is to transfer data to other apps, then that would trigger TransactionTooLargeException, since max limits are around 100KB, would need to test if it does trigger exception, not sure how browsers handle that. Rows can be increased to 50000, resulting in 4MB data, although likely a blind man may not do that.

When termux shares terminal text, it truncates the data to 100KB to prevent exceptions. I guess, when user goes into selection mode, then can start sending all the text, or after truncation.

https://www.reddit.com/r/tasker/comments/prro8t/autoshare_crashed_when_i_pasted_the_file_path/

shindere commented 2 years ago

Dave Mielke (2022/06/24 09:30 -0700):

[quoted lines by Sébastien Hinderer on 2022/06/24 at 09:18 -0700]

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

Brltty sees the whole TextView, even if all of it doesn't fit on the physical screen. What happens to be visible on the screen isn't relevant to a braille user just like what can fit on a braille display isn't relevant to a sighted user. Another way to put it is that both scroll independently and differently from one another.

I'm not sure I agree things always need to be so and can absolutely not be different.

For instance on the Linux text console, when shift-pageup and shift-pagedown were working, what was visible to the visually impaired user was exactly what was on the console and that seemed nice to me.

I realise that when using a braille display one does not always be limited to what there is on the screen of the phone, but I also think there may be cases where synchronisaiton is desirable, typically if a visually impaired user and a sighted user would like to work together.

shindere commented 2 years ago

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

agnostic-apollo commented 2 years ago

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

You may be able to do that in landscape mode, not in portrait mode since rows would be greater than columns. I guess terminal margin could be adjusted with terminal-margin-horizontal and terminal-margin-vertical properties and/or terminal zoomed to appropriate font size with pinch and zoom of fingers to get that in portrait mode too.

shindere commented 2 years ago

agnostic-apollo (2022/06/24 09:54 -0700):

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

You may be able to do that in landscape mode, not in portrait mode since rows would be greater than columns. I guess terminal margin could be adjusted with terminal-margin-horizontal and terminal-margin-vertical properties and terminal zoomed to appropriate font size with pinch and zoom of fingers to get that in portrait mode too.

Thanks. Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else. The device is called a BrailleSens 6, from HIMS.

DaveMielke commented 2 years ago

[quoted lines by Sébastien Hinderer on 2022/06/24 at 10:12 -0700]

Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else.

And may I dare to add that accessibility should never be added with just one use case in mind.

shindere commented 2 years ago

Dave Mielke (2022/06/24 10:22 -0700):

[quoted lines by Sébastien Hinderer on 2022/06/24 at 10:12 -0700]

Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else.

And may I dare to add that accessibility should never be added with just one use case in mind.

For sure. I just mentionned mine as one among others, definitely not as the only use-case.

agnostic-apollo commented 2 years ago

The device is called a BrailleSens 6, from HIMS.

You would still need to configure termux as per your rows and columns requirements so that data is sent to braille device accordingly. In future, shell APIs are planned to configure terminal font size, margins, rows, etc.

DaveMielke commented 1 year ago

[quoted lines by agnostic-apollo on 2022/06/24 at 09:54 -0700]

You may be able to do that in landscape mode, not in portrait mode since rows would be greater than columns.

I don't think the average blind user realizes how constrained a mobile device's screen actually is. The truth is that the average mobile screen makes a mockery out of someone's very expensive and much larger braille display.

Why not get out of the business of hard-coding these sizes. I suggest that you allow the user to configure the desired screen width and height. Of course, you should pick sensible defaults, but allowing them to then be changed rather than simply being hard-coded would be the way to go.

And, of course, Linux doesn't use hard-coded sizes either. They default to whatever width and height the selected font permits, but there's also the stty command which can be used to set an explicit width and/or height.

I guess terminal margin could be adjusted with terminal-margin-horizontal and terminal-margin-vertical properties and terminal zoomed to appropriate font size with pinch and zoom of fingers to get that in portrait mode too.

It's actually very difficult for a blind user to effectively use finger-based zoom adjustment. That obnly works well when one has a view of the whole screen all at once. A blind user needs to work with actual numbers.

shindere commented 1 year ago

I agree with Dave.

Ideally, I would like to be able to have a 25x80 screen as on my PC, because I don't really care about what appears on the screen, most of the time.

agnostic-apollo commented 1 year ago

Why not get out of the business of hard-coding these sizes.

The sizes are not technically hardcoded, just defaults exists. Pinch and zoom exists as you said to change the sizes. Terminal by default fills whatever the available space exists on the screen, other than keyboard if its open, just like how pretty much all android apps behave. In landscape mode, this results in columns being more than rows.

There are plans to add APIs to resize from command line, others have requested it too. I wasn't aware that blind people would prefer restricted sizes, so will keep it under consideration when implementing, specially allowing users to break aspect ratio. It won't happen in next release coming soon, but cursor position fix should be available.

shindere commented 1 year ago

agnostic-apollo (2022/10/11 02:43 -0700):

Why not get out of the business of hard-coding these sizes.

The sizes are not technically hardcoded, just defaults exists. Pinch and zoom exists as you said to change the sizes. Terminal by default fills whatever the available space exists on the screen, other than keyboard if its open, just like how pretty much all android apps behave. In landscape mode, this results in columns being more than rows.

Sure it makes sense that, by default, the size of the terminal adapts to the space available on the screen and its orientation.

It's jsut that, besides to that, it would be helpful to have a mode where the size of the terminal does not take this space and orientation of the screen into account, as odd as it may sound. and of course the size shoudln't not change if for any reason the phone is turned from landscape to portrait mode, etc.

In this context, we are in the situation where a blind suer can somehow see more than a sighted user can, which is a bit the opposite of what happens usually, even if htis is not completely true.

There are plans to add APIs to resize from command line, others have requested it too. I wasn't aware that blind people would prefer restricted sizes, so will keep it under consideration when implementing, specially allowing users to break aspect ratio.

What do you mean by "restricted"?

It won't happen in next release coming soon,

Not a big deal. Many thanks for your interest and efforts to support us.

but cursor position fix should be available.

That's really cool news. If you need some testing I can try, I'd just need to know what the most straightforward way to test, if possible without having to recompile everything.

agnostic-apollo commented 1 year ago

t would be helpful to have a mode where the size of the terminal does not take this space and orientation of the screen into account,

Yes, you are right, that will have to be taken care off too. Current, the font size is what defines the row and columns and font size is what's changed when pinch zoom is done.

What do you mean by "restricted"?

Not fill screen and use less than the available screen space.

If you need some testing I can try,

Sure, will let you know, you won't need to build, you would be able to download from github actions. But if you installed termux from F-Droid instead of Github, then will have to uninstall F-Droid version first since it has different signing key for apk.

shindere commented 1 year ago

agnostic-apollo (2022/10/11 03:14 -0700):

What do you mean by "restricted"?

Not fill screen and use less than the available screen space.

Would things really happen that way? What's the typical size of the on-screen terminal? How many rows and columns does it have, approximately?

I am astnoshed because what I imagined was that the terminal a blind person would find comfortable would be too big to fit on the screen rather than too small.

If you need some testing I can try,

Sure, will let you know, you won't need to build, you would be able to download from github actions.

Okay good. The more direct the download URL, the simpler.

But if you installed termux from F-Droid instead of Github, then will have to uninstall F-Droid version first since it has different signing key for apk.

No no, didn't do that. No termux app installed at the moment.

agnostic-apollo commented 1 year ago

Would things really happen that way?

25 rows in landscape at all times would consume less than half the display size.

How many rows and columns does it have,approximately?

Depends on device display size and font size set by user. For me, 30-40 rows with keyboard open and 60-80 rows with it closed. Cols are around 75-80.

shindere commented 1 year ago

Okay I did'nt realise so much would fit on a phone's creen!

If the typical terminal one would want as a visually impaired user fits on one screen then it's even better because we are brought back to what happens on a computer.

And, in that case, one other helpful thing to have would be a possiblity to highlight on the phone's creen the region which is currently rendered in braille (highlight braille window, for short). this is useful when a blind user and a sighted user want to work together. That way, the sighted user can know what the blind user is reading and, if he wants to show him another part of the screen, he can say things like Go up three lines.

One other context where highlighted briallle window is useful is to explian people how one reads the screen, to illustrate that we don't read it at once but rather see it one piece at a time.

Not to say it should be implemented right now, rather so the implementros have the need in mind when writing code and designing APIs.

DaveMielke commented 1 year ago

[quoted lines by agnostic-apollo on 2022/10/11 at 02:43 -0700]

I wasn't aware that blind people would prefer restricted sizes,

Some do, and some don't. This mostly has to do with which kind of braille device any given user has, and, more specifically, which kinds of controls his braille device has. I myself, for exaple, don't mind large sizes at all - my current Linux screen, for example, is 240 columns by 67 rows, and that's simply because it's what the kernel has "decided.

I think what you need is a dialog way to set the screen dimensions as zoom gestures aren't very easy to use by blind people. And, yes, unlimited should be offered as well.

DaveMielke commented 1 year ago

[quoted lines by Sébastien Hinderer on 2022/10/11 at 03:04 -0700]

the size shoudln't not change if for any reason the phone is turned from landscape to portrait mode, etc.

Yes, this is a very critical point! A sighted person, due to his full view of the screen, is aware that the screen's orientation has changed, but a blind person doesn't get this feedback. To a blind person, such a change in the screen's orientation "looks" like the braille display has lost its place.

In this context, we are in the situation where a blind suer can somehow see more than a sighted user can, which is a bit the opposite of what happens usually, even if htis is not completely true.

Actually, that's just the way it is when using Android. Brltty lets its user navigate the entire text view, no matter how far off the screen it goes in any direction. To do otherwise would actually confuse a blind user because, for example, lines would start/end at unpredictable places given that a blind user has no idea where the edges of the physical screen are. It's also far more trouble than it's worth for a blind user to have to constantly scroll the screen - he'd rather simply navigate it with the controls on his braille device.

shindere commented 1 year ago

Dave Mielke (2022/10/11 08:24 -0700):

In this context, we are in the situation where a blind suer can somehow see more than a sighted user can, which is a bit the opposite of what happens usually, even if htis is not completely true.

Actually, that's just the way it is when using Android. Brltty lets its user navigate the entire text view, no matter how far off the screen it goes in any direction. To do otherwise would actually confuse a blind user because, for example, lines would start/end at unpredictable places given that a blind user has no idea where the edges of the physical screen are. It's also far more trouble than it's worth for a blind user to have to constantly scroll the screen - he'd rather simply navigate it with the controls on his braille device.

Wouldn't it be possible to make sure the screen scrols automatically, so that the portion of text which is displayed in braille is guaranteed to always be visibleon-screen?

I do realise this does not really matter when using a phone on one's own, but as soon as one works with a sighted person it becomes of critical importance in my opinion, and it can be really frustrating when the portions of screen viewed in braille and viewed on the screen's phone become totally unrelated.

DaveMielke commented 1 year ago

[quoted lines by Sébastien Hinderer on 2022/10/11 at 08:46 -0700]

Wouldn't it be possible to make sure the screen scrols automatically, so that the portion of text which is displayed in braille is guaranteed to always be visibleon-screen?

I think it'd be rather unnerving to a sighted colleague to see the screen jumping around as the view is being navigated from the braille device. Peraps there should be a command to scroll the visible part of the view to where the braille display currently is.

shindere commented 1 year ago

Dave Mielke (2022/10/11 17:14 -0700):

[quoted lines by Sébastien Hinderer on 2022/10/11 at 08:46 -0700]

Wouldn't it be possible to make sure the screen scrols automatically, so that the portion of text which is displayed in braille is guaranteed to always be visibleon-screen?

I think it'd be rather unnerving to a sighted colleague to see the screen jumping around as the view is being navigated from the braille device.

I disagree. When working together this is an important feature. One may want to be able to turn it of, okay, but not providing it at all just because it might not be useful in all circumstances does not look right to me.

Peraps there should be a command to scroll the visible part of the view to where the braille display currently is.

To me, having to issue a command to synchronize explicitly would be a real burden. When we work together with @sthibaul we heavily rely on this synchronization happening automatically.

termux / termux-app

Let screen readers catch cursor position #2543