Make braille_previousLine and braille_nextLine functional outside of text content

[nvaccessAuto]

Reported by msuch on 2014-04-28 07:49 On Seika, Braille_previousLine is bound to b3and braille_nextLine is bound to b4. They don't work as I would expect: They work fine when in a text editor, pressing b3 gets to previous line, pressing b4 gets to next line. The problem occurs when in a list, for example on the desktop or in a menu like NVDA menu. Pressing b3 or b4 does nothing, I would expect then to bring me to the previous or next item.

[nvaccessAuto]

Comment 1 by jteh on 2014-04-28 08:51 This is not specific to Seika displays, nor is it a bug as such.

The issue is that outside of text controls, the concept of "line" is far less clear.

• Lists and menus are usually vertical, but this isn't always the case. For example, the Windows Desktop reports as a list, but it's actually arranged in multiple columns. They are both lists as far as NVDA is concerned, so it will treat them the same way.
• Then there's the question of what to do for other controls. If you're focused on a button, where should NVDA move next? The next control might not be on the next line. It might not even be focusable. Changes: Changed title from "On Seika display, braille_previousLine and braille_nextLine don't work as expected" to "Make braille_previousLine and braille_nextLine functional outside of text content"

[nvaccessAuto]

Comment 2 by msuch (in reply to comment 1) on 2014-04-28 09:53 Well, Except if I miss something important, I think that braille_previousLine and braille_nextLine should behave the same way than up and down arrows on main keyboard.Replying to jteh:

This is not specific to Seika displays, nor is it a bug as such.

The issue is that outside of text controls, the concept of "line" is far less clear.

• Lists and menus are usually vertical, but this isn't always the case. For example, the Windows Desktop reports as a list, but it's actually arranged in multiple columns. They are both lists as far as NVDA is concerned, so it will treat them the same way.
• Then there's the question of what to do for other controls. If you're focused on a button, where should NVDA move next? The next control might not be on the next line. It might not even be focusable.

[nvaccessAuto]

Comment 3 by jteh on 2014-04-30 07:05 Sometimes, down/up arrow doesn't take you to the previous line. It might move to the next control in a dialog, open a menu or change the value of a slider. Also, that doesn't work for review at all, only for focus.

[nvaccessAuto]

Comment 4 by msuch (in reply to comment 3) on 2014-05-01 11:23 Replying to jteh: I think that in lists like for example the Thunderbird's message lists, the application menus etc, these commands should act like arrows, without sppech as they do in texts, but I understand why they don't do so for now.

Sometimes, down/up arrow doesn't take you to the previous line. It might move to the next control in a dialog, open a menu or change the value of a slider. Also, that doesn't work for review at all, only for focus.

[jcsteh]

@dkager, @leonardder, @bramd, this is technically a "feature", but users tend to think of this as a deficiency rather than a nice-to-have. Thoughts on priority? Does every other screen reader do this? Any ideas on how to address the issues described above?

[bramd]

@jcsteh Well, now I think of it, I think every screenreader does at least something when pressing braille up/down.

I haven't used JAWS extensively for years, but if I remember correctly, scrolling up/down would bring you out the structured mode and into line mode. The line mode in JAWS displays what is kind of the equivalent of a line in NVDA's screen review mode. And yes, on the desktop that means that you would miss the rest of the columns on your focus line. JAWS has a toggle to link braille scrolling to the control of the cursor, so you simulate arrow keys when scrolling up/down. And yes, as sometimes also happens in NVDA's screen review, you don't see things or don't see anything at all. Back in the days the line mode and JAWS cursor worked best with old style window controls and would fail with more modern applications that needed UIA/MSAA to interact. JAWS now has something called the touch cursor to improve that situation, but I'm unsure how that works together with a braille display.

Supernova also allows scrolling up/down with a braille display and will show lines on the screen ,I'm unsure how a focus on a line with multiple columns, such as on a desktop, is handled. I think @dkager can fill us in there.

I'm unsure how/if we should handle this, but here are some considerations:

• Some braille displays have not that much keys, Handytech's BasicBraille for example has just 6 buttons and some Baum displays as well, if I remember correctly. Scroll up/down will be mapped everywhere, since it is essential functionality. Displays with more buttons will usually map simulating arrow keys as well
• Simulating arrow keys on a display might make sense for a braille only user to faster explore a list or setting a slider without moving hands from braille display to keyboard and back again
• I don't think scrolling by line, which still is assumed due to the one line design of a braille display and the old DOS days where a line mapped cleanly to a line on screen still makes sens in a graphical environment. We introduced object navigation and probably should think more in lines of that for mapping useful braille keys in this kcase

[jcsteh]

1. When tethered to review, we could just move to the next/previous object in the flattened object hierarchy as we do for flicks on touch screens. I believe we discussed this option on our call.
2. When tethered to focus, the best I can come up with is that we move using the flattened object hierarchy, but we skip everything that isn't focusable. However, that could mean skipping a huge number of objects, so this could be slow.
3. If we have auto tethering which switches to review when you move away from the focus, I guess it would just auto tether to review and act as per 1) above.

Marking as feature.

[dkager]

Yes, we discussed (1) in the last call. I'm still a big fan of this, but there are some issues that we also discussed. I'm interested in a combination of (1) and (3), assuming that flat review is reliable enough. I guess it makes a stronger cse if both touch and braille rely on it.

Scrolling through screen lines works and is even quite efficient, but only when you are used to it and when the underlying app supports it. This is where JAWS/SuperNova are struggling a lot nowadays. In physical mode, SuperNova underlines the focused item. But other than to know exactly how a screen is laid out, e.g. when instructing a sighted user or designing your own UIs, I don't think there is much benefit to physical mode anymore. I did like that feature very much in the Windows 2000/XP era though, so it's good that we discussed it. One good use for physical mode was scrolling up/down in list views (until you reach the edge of the window and it stops). For NVDA, previous/nextLine could simulate up/down arrow k eys, but then we are defining specific bindings for certain controls and there will still not be a universal approach. So maybe we could see what happens with flat review?

[LeonarddeR]

I agree with 1 and 3, with the exception that I"m not sure whether the flattened object hierarchy should be used. When moving up and down on the desktop, for example, I think it would be enough just to move through all the icons on the desktop (i.e. next/previous object), but only on that level in the object hierarchy. Flattened object review is a bit broad, and you get things you don't expect (i.e. when you are on the first desktop icon, flattened object nav brings you back to the parent list).

[dkager]

Yes, flat review is most intuitive for left/right, not up/down. But then what do you do with up/down? Simulate arrows? That isn't always appropriate.

[MarcoZehe]

Wondering if this could still go somewhere. Like this being more broadly defined as "next" and "previous" appropriate unit. In edit controls, this is lines, since that is the most obvious. In other contexts, this could mean next and previous object, so like flicking right and left on a touch screen. Routing buttons could then be used to focus/execute on that new object. I realize this would move away from the strict line concept. But as screens are so dynamic on Windows anyway, and hardly ever one line of icons on one computer matches the same set of icons on a second, unless all parameters like screen resolution etc., are exactly the same, the only place where lines actually are conceptually exactly that, are edit controls anyway, I think it would make sense to broaden the scope of next and previous. What do you all think?