search

microsoft / vscode

Visual Studio Code
https://code.visualstudio.com
MIT License
164.97k stars 29.53k forks source link

Edit Context: Screen Reader Users Feedback #229051

Open aiday-mar opened 2 months ago

aiday-mar commented 2 months ago

Pinging @jooyoungseo, @rperez030 and @meganrogge

Recently we have been working on adopting the EditContext API (https://developer.mozilla.org/en-US/docs/Web/API/EditContext_API) within VS Code. The EditContext is a new DOM property that can be set on DOM elements which decouples text input from the textual mutations of the DOM element. Essentially when the user focuses a DOM element on which an EditContext is set and types, the EditContext fires 'textupdate' events, and it is up to the user to mutate the DOM element with the changes from this event.

There are several reasons why we have adopted this API:

We have an experimental setting which enables the EditContext API with ID editor.experimentalEditContextEnabled. We would like to ask @jooyoungseo and @rperez030 if when you have time you could try the setting with a screen reader and let us know if you see any issues on typing in the various inputs of VS Code (could be the editor, the panel chat input, the SCM view, the quick input, ie any input that accepts text insertions). We would like to gather feedback from screen reader users before enabling this setting by default.

rperez030 commented 2 months ago

Thanks for the heads-up. I'll give it a try and report back here.

From: Aiday Marlen Kyzy @.> Sent: Thursday, September 19, 2024 6:55 AM To: microsoft/vscode @.> Cc: Mention @.***> Subject: [microsoft/vscode] Edit Context: Screen Reader Users Feedback (Issue #229051)

Pinging @jooyoungseohttps://github.com/jooyoungseo, @rperez030https://github.com/rperez030 and @meganroggehttps://github.com/meganrogge

Recently we have been working on adopting the EditContext API (https://developer.mozilla.org/en-US/docs/Web/API/EditContext_API) within VS Code. The EditContext is a new DOM property that can be set on DOM elements which decouples text input from the textual mutations of the DOM element. Essentially when the user focuses a DOM element on which an EditContext is set and types, the EditContext fires 'textupdate' events, and it is up to the user to mutate the DOM element with the changes from this event.

There are several reasons why we have adopted this API:

We have an experimental setting which enables the EditContext API with ID editor.experimentalEditContextEnabled. We would like to ask @jooyoungseohttps://github.com/jooyoungseo and @rperez030https://github.com/rperez030 if when you have time you could try the setting with a screen reader and let us know if you see any issues on typing in the various inputs of VS Code (could be the editor, the panel chat input, the SCM view, the quick input, ie any input that accepts text insertions). We would like to gather feedback from screen reader users before enabling this setting by default.

- Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vscode/issues/229051 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJH5KGXL4TYU3I5ZCMMZUR3ZXKUOXBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLKJTGE3DQMJTGM3DJJDOMFWWLKLIMFZV63DBMJSWZAVFOZQWY5LFVI3TIMRYGM2DKOJVGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44UQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBUGE4DQMJZGAYIFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDKMZVHE4TGMZQGWBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGMYTMOBRGMZTMNECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UNZUGI4DGNBVHE2TJJ3UOJUWOZ3FOKTGG4TFMF2GK. You are receiving this email because you were mentioned.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jooyoungseo commented 2 months ago

NVDA does not seem to support this new editor mode; word-by-word navigation with JAWS is not reliable. Sorry for the brevity. This is my initial and quick testing results.

rperez030 commented 2 months ago

Hi team,

After enabling the setting, both NVDA and Narrator no longer recognize the editor as a text editor and cannot track the cursor properly. While NVDA performs slightly better, it is still not usable.

Based on my observations, this may be due to Chromium not returning the correct roles and states through the IAccessible API.

Here are the API returns with the setting enabled and disabled:

With the Setting Enabled:

With the Setting Disabled:

I think this will require some work in Chromium to be properly supported.

aiday-mar commented 2 months ago

Hi @rperez030 and @jooyoungseo thank you both for your testing. Apologies I should have tested myself first with NVDA to catch this simple bug. I will investigate this issue further.

I was wondering where you got this accessibility information from, is it from the aViewer tool which allows to inspect accessibility information? In case you do use the aViewer tool, is it also very slow for you?

aiday-mar commented 2 months ago

Hi @rperez030 I am using NVDA with the setting editor.experimentalEditContextEnabled. I put the cursor in the editor and am using the arrow keys to read the lines one by one, this seems to work. Reading letter by letter seems to work too.

I tested with Narrator and noticed that the focus box does not follow the editor cursor. I therefore transferred some of the aria labels from the current editor implementation to the edit context, notably: role, aria-required, aria-multiline, aria-autocomplete, aria-roledescription. With these attributes the focus box follows the cursor. The PR is here: https://github.com/microsoft/vscode/pull/229361/files.

I am wondering if perhaps accessibility is fixed or partially improved in NVDA with the PR above?

When you say NVDA/Narrator no longer recognize the editor as a text editor and that it can not track the cursor properly, could you perhaps provide steps on how to reproduce this issue? Thank you.

rperez030 commented 2 months ago

Open VS code, and write the following in a new document.

This is a test.

This is a line.

This is another line.

Use the arrow keys to navigate, line by line and character by character. Observe how NVDA reports the empty lines between the text lines.

Now, enable editor.experimentalEditContextEnabled, and repeat the test.

At least in my environment, NVDA no longer tracks the cursor, and the entire document appears as a single line in the Braille display.

From: Aiday Marlen Kyzy @.> Sent: Monday, September 23, 2024 6:26 AM To: microsoft/vscode @.> Cc: Mention @.>; Comment @.> Subject: Re: [microsoft/vscode] Edit Context: Screen Reader Users Feedback (Issue #229051)

Hi @rperez030https://github.com/rperez030 I am using NVDA with the setting editor.experimentalEditContextEnabled. I put the cursor in the editor and am using the arrow keys to read the lines one by one, this seems to work. Reading letter by letter seems to work too.

Could I ask you what you mean exactly by NVDA no longer recognizes the editor as a text editor and that it can not track the cursor properly, and potentially provide steps to reproduce this issue?

- Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vscode/issues/229051#issuecomment-2367806111 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJH5KGQUF7SA7C36LEWBL2TZX7UD5BFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZTCNRYGEZTGNRUURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UNZUGI4DGNBVHE2TJJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOKIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2DCOBYGE4TAMECUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRVGM2TSOJTGMYDLAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTGMJWHAYTGMZWGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKG42DEOBTGQ2TSNJUU52HE2LHM5SXFJTDOJSWC5DF. You are receiving this email because you were mentioned.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

aiday-mar commented 2 months ago

Hi @rperez030 thank you for the clarification. Today I am testing the Edit Context with NVDA with the recent aria attributes changes.

After setting the setting to true, reloading the window, and after some (I assume notification) text is read, I place the cursor at the top of the editor and hear the following as I move the cursor down:

This is a test, Blank, This is a line, Blank, This is another line

This seems to correspond to what is read by NVDA without the setting.

The text that is read however does not appear in the NVDA speech viewer, nor do I see it in the Braille Viewer. I tested with the setting turned off, and the text does not appear there either. I am not sure if I am missing some config to see the text that would be sent to a braille device.

Please let me know if I am missing or misunderstanding something.

rperez030 commented 2 months ago

In theory, everything that gets sent to the speech subsystem should also be available in the speech viewer. The same applies to braille. does the speech viewer even work for you outside VS Code? Happy to get on a call to troubleshoot if that helps speed things up.

my schedule is quite busy today, but we could connect tomorrow or friday.

aiday-mar commented 2 months ago

Hi @rperez030 thank you for suggesting. Today I will not be able to, but tomorrow or after tomorrow I can. What time would suit you? Thanks again.

aiday-mar commented 2 months ago

Hi @rperez030. After updating VS Code I am able to see the correct text read out in the speech viewer on NVDA and the braille display. The blank lines between the text are read out by NVDA as blank. Please let me know if you notice other accessibility issues in the editor with the experimental setting.

jooyoungseo commented 2 months ago

@aiday-mar -- Have you tested word-by-word navigation thoroughly? When I tested the following with ctrl+LeftArrow and ctrl+RightArrow multiple times, the cursor tracking is not mapped with what is being heard.

The quick brown fox jumps over the lazy dog.

  1. Press Home to place your cursor at the beginning of this sentence.

  2. Press ctrl+RightArrow until you hear "brown"

  3. Press ctrl+LeftArrow to move one word to the left and you will hear "quick"

  4. Press ctrl+RightArrow and you will hear "brown" again

  5. In the word saying "brown," use LeftArrow or RightArrow to move your cursor character-by-character

aiday-mar commented 2 months ago

Hi @jooyoungseo thanks for the steps, I have not tested this case. I will investigate the issue with these steps.

Edit: It would seem from the initial investigation that with the setting disabled control+Left and control+right moves the editor cursor. This does not happen with the setting enabled, so it could be a keybinding issue. Looking some more.

aiday-mar commented 2 months ago

Hi everyone, I have investigated the issue and I have traced this issue to the fact that we don't update the editor cursor when the EditContext dom node selection is changed. When you navigate from word to word, the selection changes inside of the dom node, and this has to be propagated to the editor. The following PR should fix this: https://github.com/microsoft/vscode/pull/230204.

rperez030 commented 2 months ago

I have also identified an issue related to blank lines. When reading a file line by line with NVDA, it appears that NVDA does not correctly identify empty lines. I observed that the Unicode character generated by the Enter key varies depending on whether the setting is enabled.

With the setting disabled, the Unicode output is: "10,0 x a".

With the setting enabled, the Unicode output is: "13,0 x d".

aiday-mar commented 2 months ago

Ah yes I see, so in one case it is a Line Feed character and in the other case a Carriage Return character. I will look into this.

aiday-mar commented 2 months ago

Hi after some investigation, I discovered this was happening because of the recent change I did to fix the selection-change issue. My fix introduced a new issue in that the text for the screen readers is not correctly aligned with the cursor position. Essentially there was an offset issue, which I fix with https://github.com/microsoft/vscode/pull/230475.