search

microsoft / terminal

The new Windows Terminal and the original Windows console host, all in the same place!
MIT License
95.94k stars 8.35k forks source link

Jaws screen reader #11730

Closed gomaltao closed 2 years ago

gomaltao commented 3 years ago

Windows Terminal seems to be totally inaccessible with the JAWS screen reader.

zadjii-msft commented 3 years ago

Which version of the Terminal? Which OS version? Which JAWS version? I'm pretty sure JAWS is supposed to work... /cc @carlos-zamora

carlos-zamora commented 3 years ago

@codeofdusk I know you use NVDA a lot, have you tried using JAWS? Or have you heard from any JAWS users trying to use Windows Terminal at all?

codeofdusk commented 3 years ago

Sorry, no.

@grahamwp What happens if you try and run wt from the run dialog on your machine? Is the opened terminal accessible at all?

gomaltao commented 3 years ago

Windows Terminal 1.11.2921.0 OS Name: Microsoft Windows 10 Pro OS Version: 10.0.19042 N/A Build 19042 JAWS Version 2020.2012.13 ILM

I have tried to use WT since the first beta without any success with JAWS.

gomaltao commented 3 years ago

I closed the issue by misstake.

Grahamwp commented 3 years ago

I can confirm this is the case with Windows Terminal version 1.11.2921.0and the latest version of JAWS, 2022.2110.36.400.

zadjii-msft commented 3 years ago

I'm kinda shocked that JAWS doesn't work at all. We should fix that. Does the vintage console host work? If it does, then it shouldn't be too hard to find the piece of console code we're missing in the Terminal that makes JAWS work.

codeofdusk commented 3 years ago

This doesn't surprise me as i highly suspect that JAWS uses the legacy APIs. (only Narrator uses UIA by default at this point, though I'm pushing for NV Access to do so in builds of conhost with the #6986 fix). NVDA exclusively uses UIA in wt though.

Grahamwp commented 3 years ago

Late reply (I tried to reply by email earlier but it didn't work): Assuming you mean cmd.exe, yes, JAWS does work in that. It also workes in Power Shell.

zadjii-msft commented 3 years ago

@codeofdusk Is there any way for us to confirm your suspicion, that they use the legacy APIs? It might be time for us to encourage them to move on to the newer APIs 😄 I'm not sure they have a public issue tracker anywhere...

gomaltao commented 3 years ago

As a senior .net developer I need to work in powershell a lot. Windows terminal would improve my working conditions a lot. I would really like to know where the problem is. JAWS has problem with announcing breakpoints in Visual Studio for several generations of VS ide. Again it would be good to know if the problem is with the screen reader or VS ide.

codeofdusk commented 3 years ago

Maybe @carlos-zamora could do a build of conhost that shows dummy text in the UIA buffer in place of the actual text. If we get his text then they're using UIA, else they're using legacy. (for whatever it's worth I highly doubt they're using UIA for much if at all).

I haven't been an active JAWS user for about ten years, but @Grahamwp can try builds out for us and report back. I also suspect that they won't be amenable to making changes quickly if at all.

carlos-zamora commented 3 years ago

Maybe @carlos-zamora could do a build of conhost...

This issue is tracking Windows Terminal, not conhost. Like you said earlier, since Windows Terminal only has UIA (as in, there's no legacy API option), a screen reader needs to use UIA here.

When I was developing Windows Terminal's UIA patterns, NVDA and Narrator both "worked". Like, they were testable and I could clearly see when I made a mistake. I'd be surprised if JAWS saw Windows Terminal and decided to do nothing (because in its eyes, it would see that it's a random app that exposes UIA, and I assume like any screen reader, it would leverage that and try to use it).

I'll get in contact with JAWS and see what we can do, but I'm pretty sure the issue here is on their end. I'll report back when I get more news 🙂

DHowett commented 3 years ago

@carlos-zamora I think @codeofdusk's suggestion is to make a "trojan horse" build of conhost that gives different answers for Legacy and Modern accessibility. Since it's our only product that currently supports both forms of accessibility, it would be an easy conversion. We could use that "trojan horse" build to figure out which accessibility implementation JAWS is using.

carlos-zamora commented 3 years ago

Ok. Here's a version where any time you get text from the buffer, it'll be "Fake Data from UIA". ConHost - UIA with fake data.zip

codeofdusk commented 3 years ago

@carlos-zamora that's a zipped lnk pointing at D:\Terminal\bin\x64\Release\OpenConsole.exe.

carlos-zamora commented 3 years ago

Derp... here's the actual exe. ConHost - UIA with fake data.zip

codeofdusk commented 3 years ago

For some reason, Windows Defender is flagging that file and I can't figure out how to allow it.

carlos-zamora commented 3 years ago

Ok, try this one then. It's signed. conhost-x64-release.zip

codeofdusk commented 3 years ago

Thanks @carlos-zamora .

@Grahamwp please do the following on your machine, ping me on Discord if you run into trouble:

  1. Download the last build from @carlos-zamora above (this one).
  2. Unzip and run OpenConsole.exe.
  3. Try to review the window by line.

Do you get normal output (Microsoft Windows blahblah and a prompt) or "fake data from UIA" over and over again? Or something else?

codeofdusk commented 3 years ago

Also try autoread and the virtual buffer if JAWS still has that.

Grahamwp commented 3 years ago

I downloaded the file and I just get the normal Windows console output, whether I use autoread, review the window with the JAWS cursor, or try the virtual buffer.

codeofdusk commented 3 years ago

As expected, they aren't using UIA at all in consoles, so supporting terminal will be a big task.

zadjii-msft commented 3 years ago

Well this is good to have some confirmation and get this loop closed! JAWS is using the legacy accessibility APIs, and they're going to need to move to the modern ones to make the Terminal work with JAWS. UIA has been around for some 15 years now, so I'm a little surprised they haven't added support by now.

Thanks all!

notes found while writing: Microsoft Active Accessibility and UI Automation Compared

codeofdusk commented 3 years ago

Jaws does support UIA in some apps, but not conhost (similar to NVDA's default behaviour before nvaccess/nvda#10964).

Maybe @dglee42 or @robgallo might know how to attach JAWS terminal support to UIA events via a script (if it even works that way)?

Grahamwp commented 3 years ago

There's also relevant documentation from Freedom Scientific but I wouldn't personally be able to further help with this on the JAWS scripting side.

carlos-zamora commented 3 years ago

FYI, I've managed to reach out to JAWS. Hoping I can provide some direct support to them. They are definitely aware of this issue and I've even forwarded them this GitHub issue. 🙂

zadjii-msft commented 2 years ago

Moving this into 1.14 based on recent conversations.

zadjii-msft commented 2 years ago

@carlos-zamora should have fixed this in #12358, and confirmed that JAWS made the fix on their end as well. We're not sure exactly which version of JAWS the fix is in, but it should be in the next version of the Terminal.