API: expose interrupt-parameter for outputMessage

[mespotine]

I posted about this in another issue, but it fits better in its own issue.

I looked through the code and noticed, that in reaper_osara.cpp, outputMessage has an additional boolean-parameter interrupt, which looks like it could interrupt the currently read message of TTS. I would like to ask for exposure of this parameter to osara_outputMessage.

My usecase is the following: In my GuiLib, I'm trying to mimick the behavior, that hovering above a ui-element, tts is reading out the label/text/etc of the ui-element. Usually it works like: you hover over one ui-element, tts is reading the caption out. When I move the mouse now to another ui-element while the reading is still happening, tts gets interrupted and reads the caption of the new ui-element.

Now here's the problem: osara_OutputMessage will always read out the full message before reading the next one. So if I hover over three ui-elements, one after another, osara will read out the entire caption of all three of them one by one, which takes ages. I would like to be able to interrupt and read immediately the next caption to the user for a much more fluent user experience.

I currently try to cheat me through by just feeding the first 20 characters of the caption/text/etc to osara so the read out message isn't too long but that's not right.

So, if the parameter interrupt means interrupting the currently read out message, it would help me tons, if it would be available. This is the last bit I need to finish up the gui-lib entirely.

[ScottChesworth]

outputMessage is already interrupting here when I either press Control to silence speech, or when I press tab/some other keystroke to move focus in a way that results in a new report. I noticed you use the word hover, wonder whether the behaviour is different because you're using a mouse to focus there? Can you try hitting Control to confirm whether that interrupts?

[mespotine]

Cmd/Ctrl interrupts properly. But when sending two messages after another using the api-function osara_outputMessage , the first is read fully, then the second. The hovering I noticed when moving the mouse across different apps, including Reaper. It's then telling me via tts, what is underneath it, using shorter messages. Like the caption of a button or the text of an inputbox, though without an indicator that it's a button, inputbox, etc. It's probably to help using mouse in ui without keyboard usage. The focusing I already managed to code, including tab/Shift+Tab. That works properly already in my implementation. It's just the hovering, that's not working completely due me not being able to interrupt the message. I tested it again with NVDA. --------

[ScottChesworth]

I tested contiguous messages before replying earlier on, and interruption seems to be working as expected here on Win10 with NVDA, JAWS and Narrator. Are you testing on Mac there? Methinks this might be a VoiceOver specific thing.

[jcsteh]

The exported osara_outputMessage function always sets interrupt to true; you can't set it to false. In terms of implementation, osara_outputMessage calls outputMessage with only one parameter, message. The second (interrupt) takes its default value of true.

On Mac, the interrupt parameter does nothing. We already set the priority to high for those announcements, which I thought was enough to cause interrupts, but I guess not.

[mespotine]

Hmm, I tested it on Windows 7 and it never interrupts. Weird.I can work out some test code that shows the behavior on my system, if you want...

Edit: I wonder if it's just some obscure setting in NVDA that's different on my machine from yours, which affects the output behavior. The message-log of NVDA shows the message immediately, it just doesn't read it immediately...

[ScottChesworth]

Seems like reporting is happening via MSAA instead of UIA on your machine. We support both, but MSAA is only there as a fallback nowadays as its long since been deprecated by Microsoft. Weird thing is, although I could get appended reports to happen here when I force reporting through MSAA, when I test interruptibility with reports being generated by Shortcut Help, it only happens with some key bindings. The main QWERTY layout and NumPad both interrupt, but bindings on supplemental keys (media keys, app keys etc) always append instead of interrupting). Do you have Win10/11 and a recent release of NVDA there to test on? If not, give us that sample code, let's see how it performs in Win10/Win11 with UIA (that's how most screen reader users will encounter it). My thinking is - although there might be an edge case to try and shore up, if we know what you've got works well in those conditions, at least you can carry on with what you're working toward while we look into whether there's anything we can do here.

[jcsteh]

Weird thing is, although I could get appended reports to happen here when I force reporting through MSAA, when I test interruptibility with reports being generated by Shortcut Help, it only happens with some key bindings. The main QWERTY layout and NumPad both interrupt, but bindings on supplemental keys (media keys, app keys etc) always append instead of interrupting).

That interruption is NVDA, not OSARA. NVDA interrupts any speech (OSARA or otherwise) whenever you press most keys. Media keys are an exception.

My thinking is - although there might be an edge case to try and shore up,

Not quite sure what you mean here, but if the user isn't running Windows 10 Fall Creators Update or later, there's nothing we can reasonably do to get interruption working.

[ScottChesworth]

NVDA interrupts any speech (OSARA or otherwise) whenever you press most keys. Media keys are an exception.

Aha thanks, that would explain what I just observed.

if the user isn't running Windows 10 Fall Creators Update or later, there's nothing we can reasonably do to get interruption working.

Gotcha. I'd still flippin' love to know why the priority that's been set doesn't work on Mac. I mean, if @mespotine does get a GuiLib with accessibility support over the line, that's really gonna slice into the Mac user experience.

[jcsteh]

Maybe messages at the same priority don't interrupt each other; i.e. high priority only interrupts lower priority. If that's the case, nfi what to do about it.

[mespotine]

So just that I understood correctly: on Mac, the interruption of messages doesn't work? Haven't checked any mac-related stuff yet. And how often does it happen, that an app or the os sends a high priority message? Will check the Windows issue. Hope my Windows 10 installation still works. If it would work on Windows 10, I would restrict the gui-lib to Win10 and higher.

Edit: I wonder how NVDA managed to interrupt speech when hitting keys. They seem to have found a way,even on Windows 7...

[pitermach]

OSARA messages appear to interrupt here on Mac (tested by enabling control surface changes and then turning some knobs with a Komplete Kontrol instance).

I agree that if the internal function has this interrupt parameter, it might as well be exposed for scripts too. Mouse echo is one use case but I can also see it coming in useful IE if someone makes some kind of interface controlled through an external device like a control surface or joystick

[jcsteh]

Again, the API exposed osara_outputMessage already interrupts. You can't disable interruption though. If that's really needed, we'll need to add a separate function, but I'm not seeing a use case for that just now.

[ScottChesworth]

Well yeah, OSARA interupts itself on Mac, but so far as I'm aware we still have the bottleneck issue where our reports won't interupt a stream of reports from other sources in VoiceOver's buffer. That's obviously not what's happening here because Mac hasn't been on the test bench with this GuiLib project yet, just mentioning it because I think it's important that mainstream devs like @mespotine are aware of what to expect. It would be easy for them to end up chasing their tails.

On that, is it perhaps worth adding a short "Notes for developers" section to the ReadMe?

[jcsteh]

Yeah, i was thinking of adding a section a while ago that included info on the various API calls that OSARA exports, but I never got around to it. That would be the ideal place to include something like this.

The section title should probably make it clear that the section is targeted at developers of other REAPER extensions/scripts, not OSARA itself. :)

[ScottChesworth]

I'd offer to write it, but somehow I didn't know we exported any others!

[jcsteh]

The only other one we export is osara_isShortcutHelpEnabled, which was done specifically for SWS. However, we only expose it to C++ extensions, not ReaScript. Exposing it to ReaScript is simple enough, but I didn't see a reason to do that.

[mespotine]

What does osara_isShortcutHelpEnabled do? Yes, I'm also interested in such a section about how to approach development with Osara's features. Accessibility is full of things that can be done wrong so any hint is very welcome. In regards of highest-priority messages can not be interrupted, even if sent by Osara, I may have an idea to circumvent this to some degree(if you didn't already try). You could send two messages in Osara:1: empty message of highest priority. This should interrupt all messages of lower priority. 2: the actual message with slightly less priority. That way, the actual message can always be interrupted by Osara, since it's not highesr priority. And the highest-priority empty message would be short enough to not take much time and saying nothing at the same time. This works only, if TTS isn't saying "empty" when provided with a string of no characters. It wouldn't guarantee interruption, especially when other apps or the os send highest priority messages, but it would make it much more likely to interrupt the last osara-message. But if the os sends highest-priority messages, it's probably more important anyway, than most messages in Reaper, I guess. 

[jcsteh]

What does osara_isShortcutHelpEnabled do?

OSARA has a feature called shortcut help. When enabled, it intercepts all actions and just reports their name. This serves as a keyboard learning/memorisation tool, as well as helping users when they kinda remember the keyboard shortcut but not quite.

osara_isShortcutHelpEnabled returns true when this is enabled. It shouldn't be necessary most of the time, but it was needed by SWS because REAPER hooks are run in the order in which they are registered and OSARA might be loaded after other extensions. We wanted SWS actions to work with shortcut help, so we needed a way for SWS to know to let the action through the hook in that case.

In regards of highest-priority messages can not be interrupted, even if sent by Osara, I may have an idea to circumvent this to some degree(if you didn't already try).

My understanding is that this failure to interrupt occurs with messages coming from elsewhere, not from OSARA. That is, OSARA can normally interrupt its own messages, but even if they're at high priority, they sometimes can't interrupt other messages not from OSARA and thus things get stuck. I guess anything is worth a try, but given that OSARA can't even interrupt this flow of messages with a high priority message right now, I don't think sending a high priority message and then a lower priority message would do the trick. That said, I don't use Mac, so I think a Mac user with some dev skills will need to dig into this.

[jcsteh]

As noted above, osara_outputMessage already interrupts where possible. I'm not seeing a use case for adding an argument to disable interruption, but this can be reopened if a use case arises.

See also #961.