Use sound effects to improve accessibility

[isidorn]

Some of our current features are not accessible, and could use sound effects. For example the global progress. We probably do not have any electron API for sound effects so we would have to use some node module if we ever watned to do this.

Feedback from user:

I do still wish VS Code made use of sounds for stuff, like when I reach a line with an error on it, and it could use sounds to allow word-wrap to work fine, and just play a small sound when the next logical line is reached. But that's just me, and I think it should definitely be configurable. Of course, I write human-language text most of the time, and a little HTML in Markdown files, so actual programmers that are far better than me may not really care about any of this, but VS Code can definitely be used for more than programming, so I thought I'd represent that a little.

[LeonarddeR]

If global progress is a progress bar, a screen reader should be able to read it and provide updates itself. NVDA can expose progress bars with both speech and tones.

[isidorn]

@leonardder That's a good idea. So if we set the prograssbar role NVDA should handle everything on it's own? Even if that element does not have focus?

I plan to make the changes in our Progress bar for that. fyi @bpasero since I saw you made the most changes in the progress bar, I am not sure who is the owner.

@joanmarie I wonder if Orca also supports this?

[joanmarie]

@isidorn Orca has progress bar beep support, which is disabled by default, and speech + braille support which is enabled by default. If you have something for me to test, please let me know. Thanks for asking!

[devinprater]

I wonder if sounds could be used for other things, like when an action completes? For example, currently, I can run a linter on code, or something like the grammarly extension on text, but I don't know when exactly it finishes, so I have to just hit F8 until something happens, or open the problems menu.

[isidorn]

@devinprater ideally both long running actions and linter would use the progress bar. So ideally if we start tackling the progress bar to make it sound accessible it should have an improvment in those areas.

[devinprater]

Okay, yes, if it uses the progress bar, then NVDA and other screen readers should pick it up, as long as it's on the screen.

[jvesouza]

In my opinion it would be great to hear a sound when the cursor reaches a line that contains an error or a breakpoint.

[isidorn]

@joanmarie @leonardder I have pushed a first version of our progress bar using the progressbar role (I set it on the monaco-progress-container HTML element). You can try it out and let me know how it works for you. I tried it out with Orca and could not hear anything.

Easy way to trigger progress in VS Code:

• Open Git Viewlet and press the refresh button
• Open some Large workspace and open Search viewlet and search for some general term, for example 'a'. Also inlcude node_modules to make the search even slower

@jvesouza good idea. We can look into that as step 2, since what we currently want to do is to leave the sound producing part to the screen reader, and VS Code would just give a hint to the screen reader to produce sounds. @joanmarie is there maybe some other way to hint to screen readers that a sounds should be produced?

[devinprater]

Is there a way to have Narrator produce a specific sound? Maybe that could be a starting point for having these kinds of things work. NVDA does have a "spelling error" sound that maybe could be made to play on a line.

[isidorn]

@devinprater I do not know about Narrator. Feedback that we got from all our Windows users is that they use NVDA for programming work, and Narrator not that much.

We mostly focus on Orca and NVDA simply because authors of those screen readers are quite active on GitHub as you can already see on this issue :)

[devinprater]

Yeah, I just thought that since Narrator is another Microsoft product, that better integration between Narrator and VS Code could be made to test these kinds of interface sounds and such.

[isidorn]

@joanmarie friendly ping for this, when you find time to maybe try it out with Orca. I was unsucessfull as per my comment above.

[joanmarie]

So I'm seeing an accessibility event for the addition of the progressbar, but not seeing any events for the value changing. Is this working in NVDA?

[LeonarddeR]

I'm not hearing any progress updates either with NVDA.

@isidorn I assume you know about https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_progressbar_role?

[joanmarie]

@isidorn Orca presents this example via speech in Chrome/Chromium, using Orca's default settings:

<!DOCTYPE html>
<html>
<style>
#bar {
  width: 50%;
  background-color: lightgray;
}

#progress {
  width: 0px;
  height: 20px;
  background-color: darkblue;
}
</style>
<body>
<div id="bar">
  <div id="progress" role="progressbar" aria-valuemin="0" aria-valuemax="100"></div>
</div>
<button onclick="update()">Start</button>
<script>
function update() {
  var progress = document.getElementById("progress");
  var width = 0;
  var updater = setInterval(function() {
      if (width >= 100) {
          clearInterval(updater);
      } else {
          width++;
          progress.style.width = width + "%";
          progress.setAttribute("aria-valuenow", width);
    }
  }, 1000);
}
</script>
</body>
</html> 

Note that Orca's default progress bar announcement interval is 10 seconds. Users can, of course, change that. But it occurs to me that if you're not hearing the updates you expect -- and you are updating the value of aria-valuenow correctly -- another possibility might be the default setting is greater than the duration of the VSCode progressbar.

HTH!

[isidorn]

@joanmarie @leonardder thanks a lot for trying it out! It seems like Orca would not read the progress if aria-valuemin is not set. And I was not setting it since the aria docs says it is assumed to be zero if not set. Now I set it, wait for more than 10 seconds and Orca nicely reads the progress!

Thanks for help, closing this since the progress bar is now accessible.

As for sounds in other situations please feel free to file new feature requests and we can continue the discussion tehre.

[isidorn]

Verifier: have a long running progress (longer than 10s), turn on Orca on linux and make sure Orca reports progress

Here's an example how to report long running progress using the ProgressService

progressService.withProgress({ title: 'hello', location: ProgressLocation.Notification, total: 100 }, progressStep => {
            setInterval(() => progressStep.report({ message: 'increase', increment: 1, total: 100 }), 500);
            return async.timeout(100000);
        });

[isidorn]

@joanmarie in VS Code quite often we do not know how long a progress will last, those we call infinte progresses and for them we can not really report what is the increment and thus can not set aria-valuenow we only know once they are done. Visually we show it as an infinite spinner that goes in circle. Problem I see is that Orca reads nothing for these infinte progresses. It seems like Orca requires the aria-valuenow to be set in order to read anything.

I think many apps have these infinite progress and it would make sense if orca just read Progress and confirmed every 10s or so that the progress is still ongoing.

Let me know what you think.

[joanmarie]

In theory that would work. In practice, Orca responds to value-change events (discarding those which are "too soon"). If the value doesn't change.... :woman_shrugging:

I am about to go on PTO for two weeks. But if you can make me a stand-alone test case (i.e. an HTML file that I can load in Firefox and Chromium), I'll see what events, etc. I get and see if I can come up with a solution.

[LeonarddeR]

I wonder whether it could help to set aria-busy on the content that is waiting for a progress update

[joanmarie]

It depends what the goal is: An every-x-seconds announcement like is done with a progress bar, or an announcement that something is busy (similar to the "loading please wait" ... "finished loading" done on static documents). I'm happy to change Orca to do the loading message on things other than documents, but not every x seconds.

[isidorn]

@joanmarie here's the standalone sample: I just modified yours to not set aria-valuenow and to clear the role once done and it is exactly what happens in VS Code for infinite progress.

<!DOCTYPE html>
<html>
<style>
#bar {
  width: 50%;
  background-color: lightgray;
}

#progress {
  width: 0px;
  height: 20px;
  background-color: darkblue;
}
</style>
<body>
<div id="bar">
  <div id="progress" role="progressbar" aria-valuemin="0" aria-valuemax="100"></div>
</div>
<button onclick="update()">Start</button>
<script>
function update() {
  var progress = document.getElementById("progress");
  var width = 0;
  var updater = setInterval(function() {
      if (width >= 100) {
          clearInterval(updater);
          progress.attributes.removeNamedItem('role');
      } else {
          width++;
          progress.style.width = width + "%";
    }
  }, 1000);
}
</script>
</body>
</html> 

Alternative idea: would i help if vscode sets aria-valuenow to a 100 when done. Here's a sample for that (please note that it can happen that once done we immediatly after that remove the element from DOM)

<!DOCTYPE html>
<html>
<style>
#bar {
  width: 50%;
  background-color: lightgray;
}

#progress {
  width: 0px;
  height: 20px;
  background-color: darkblue;
}
</style>
<body>
<div id="bar">
  <div id="progress" role="progressbar" aria-valuemin="0" aria-valuemax="100"></div>
</div>
<button onclick="update()">Start</button>
<script>
function update() {
  var progress = document.getElementById("progress");
  var width = 0;
  var updater = setInterval(function() {
      if (width >= 100) {
          clearInterval(updater);
          progress.setAttribute("aria-valuenow", '100');          
      } else {
          width++;
          progress.style.width = width + "%";
    }
  }, 1000);
}
</script>
</body>
</html> 

[meganrogge]

@isidorn I do not hear this working #209661. I plan to investigate in April. Is it working for you?

[isidorn]

@meganrogge no. This does not work on my macOS with VoiceOver. If I remember correctly, I am not sure if this was ever working on non-Linux platforms. Since not all Screen Readers support reading progress. It would be good if you double check on Windows.

[meganrogge]

It doesn't work on windows either. Do you think we can do anything about this or is it up to the screen readers?

[isidorn]

We should decide if we: 1) Want to wait for screen readers to start supporting reading loading sounds 2) Want to play sounds to mimic the loading sound whenever there is something in progress. If we do this we should remove the progress html attributes

Let me know what you prefer.

[meganrogge]

I think probably case 2, I will explore it this iteration https://github.com/microsoft/vscode/issues/209661

[isidorn]

Sounds good to me. Let's try it out.