search

LeonarddeR / rdAccess

Remote Desktop Accessibility for the NVDA screen reader
GNU General Public License v2.0
10 stars 1 forks source link

Feature request: support playing audio feedback. #9

Closed cary-rowen closed 1 year ago

cary-rowen commented 1 year ago

On the server, using nvda+space to switch browse mode or focus mode has no sound.

Steps to reproduce

  1. Open a browser and load any web page.
  2. After the web page is loaded, press NVDA +Space to switch focus or browse mode.

    actual performance

    NVDA has no audio feedback and the user does not know which mode is currently in.

    expected performance

    NVDA provides audio feedback as expected.

LeonarddeR commented 1 year ago

Are you using an installed or a portable copy on both the server and client?

cary-rowen commented 1 year ago

I use the installed version on both server and client: Here is the log snippet:

IO - speech.speech.speak (15:59:48.451) - MainThread (1572):
Speaking ['日志片段开始点已标记，再按一次复制到剪贴板']
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (15:59:49.531) - remoteSynthDriver_2 (8380):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x052884B0> to process command 120
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (15:59:49.532) - remoteSynthDriver_2 (8380):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x04E55630> for command <SpeechCommand.INDEX_REACHED: 120>
IO - inputCore.InputManager.executeGesture (15:59:50.009) - winInputHook (7108):
Input: kb(laptop):NVDA+space
DEBUG - nvwave.playWaveFile (15:59:50.107) - MainThread (1572):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - inputCore.InputManager.executeGesture (15:59:50.935) - winInputHook (7108):
Input: kb(laptop):NVDA+space
DEBUG - nvwave.playWaveFile (15:59:50.941) - MainThread (1572):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - inputCore.InputManager.executeGesture (15:59:51.946) - winInputHook (7108):
Input: kb(laptop):shift+control+NVDA+f1
cary-rowen commented 1 year ago

Both ESpeak and WorldVoice can reproduce this

LeonarddeR commented 1 year ago

Could you please try this build on the remote system? It will not fix the issue but adds extra logging. Note that the Playing wave file canceled debug log is normal, that just means that the wave file playback is suppressed for playback on the server.

cary-rowen commented 1 year ago

I tested it with the build you provided, here is the log snippet:

IO - speech.speech.speak (19:00:30.024) - MainThread (2096):
Speaking ['日志片段开始点已标记，再按一次复制到剪贴板']
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (19:00:31.131) - remoteSynthDriver_2 (1624):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0BD3CBB0> to process command 120
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (19:00:31.134) - remoteSynthDriver_2 (1624):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x04D21750> for command <SpeechCommand.INDEX_REACHED: 120>
IO - inputCore.InputManager.executeGesture (19:00:31.931) - winInputHook (2448):
Input: kb(laptop):NVDA+space
DEBUG - external:synthDrivers.remote.remoteSynthDriver.handle_decidePlayWaveFile (19:00:31.973) - MainThread (2096):
Sending PLAY_WAVE_FILE command: {'fileName': 'waves\\focusMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
DEBUG - nvwave.playWaveFile (19:00:31.973) - MainThread (2096):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - speech.speech.speak (19:00:32.133) - MainThread (2096):
Speaking ['Gmail （打开新标签页）', '链接', CancellableSpeech (still valid)]
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (19:00:32.601) - remoteSynthDriver_3 (8064):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0BD3CBB0> to process command 120
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (19:00:32.601) - remoteSynthDriver_3 (8064):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x04D21750> for command <SpeechCommand.INDEX_REACHED: 120>
IO - inputCore.InputManager.executeGesture (19:00:32.938) - winInputHook (2448):
Input: kb(laptop):NVDA+space
DEBUG - external:synthDrivers.remote.remoteSynthDriver.handle_decidePlayWaveFile (19:00:32.944) - MainThread (2096):
Sending PLAY_WAVE_FILE command: {'fileName': 'waves\\browseMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
DEBUG - nvwave.playWaveFile (19:00:32.945) - MainThread (2096):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - inputCore.InputManager.executeGesture (19:00:35.133) - winInputHook (2448):
Input: kb(laptop):shift+control+NVDA+f1
LeonarddeR commented 1 year ago

This works as expected. Could you also provide a client log at the time server toggles browse mode?

cary-rowen commented 1 year ago

Could you also provide a client log at the time server toggles browse mode?


DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (00:38:35.928) - RemoteSpeechHandler_3 (14672):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0F256DB0> to process command 67
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (00:38:35.928) - RemoteSpeechHandler_3 (14672):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0ED2FDB0> for command <SpeechCommand.CANCEL: 67>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (00:38:35.958) - MainThread (15908):
Executing <bound method SynthDriver.cancel of SynthDriver('WorldVoice')>((), {}) on main thread
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (00:38:36.516) - RemoteSpeechHandler_2 (13548):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0F256DB0> to process command 87
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (00:38:36.517) - RemoteSpeechHandler_2 (13548):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0ED2FD50> for command <SpeechCommand.PLAY_WAVE_FILE: 87>
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (00:38:37.727) - RemoteSpeechHandler_3 (14672):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0F256DB0> to process command 87
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (00:38:37.728) - RemoteSpeechHandler_3 (14672):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0ED2FD50> for command <SpeechCommand.PLAY_WAVE_FILE: 87>
LeonarddeR commented 1 year ago

I have made some additional changes in this build, which should be installed on the client, the server is OK. Let me know how it goes.

cary-rowen commented 1 year ago

Hi, this is the log on the server, I still can't hear the sound of switching browse mode/focus mode:

IO - speech.speech.speak (01:28:27.038) - MainThread (1060):
Speaking ['日志片段开始点已标记，再按一次复制到剪贴板']
IO - inputCore.InputManager.executeGesture (01:28:28.396) - winInputHook (5440):
Input: kb(laptop):NVDA+space
DEBUG - external:synthDrivers.remote.remoteSynthDriver.handle_decidePlayWaveFile (01:28:28.420) - MainThread (1060):
Sending PLAY_WAVE_FILE command: {'fileName': 'waves\\focusMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
DEBUG - nvwave.playWaveFile (01:28:28.420) - MainThread (1060):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - inputCore.InputManager.executeGesture (01:28:29.036) - winInputHook (5440):
Input: kb(laptop):NVDA+space
DEBUG - external:synthDrivers.remote.remoteSynthDriver.handle_decidePlayWaveFile (01:28:29.039) - MainThread (1060):
Sending PLAY_WAVE_FILE command: {'fileName': 'waves\\browseMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
DEBUG - nvwave.playWaveFile (01:28:29.039) - MainThread (1060):
Playing wave file canceled by handler registered to decide_playWaveFile extension point
IO - inputCore.InputManager.executeGesture (01:28:31.593) - winInputHook (5440):
Input: kb(laptop):shift+control+NVDA+f1
cary-rowen commented 1 year ago

I installed the latest build you provided on both the server and the client, and I tested it with Windows OneCore and worldVoice. Here is the client's log:

IO - inputCore.InputManager.executeGesture (01:47:18.662) - winInputHook (19356):
Input: kb(laptop):alt+control+break
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:20.369) - RemoteSpeechHandler_0 (21164):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 83
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:20.369) - RemoteSpeechHandler_0 (21164):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BD50> for command <SpeechCommand.SPEAK: 83>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:20.387) - MainThread (6624):
Executing <bound method SynthDriver.speak of SynthDriver('WorldVoice')>((['开始  ', '按钮  ', IndexCommand(10450)],), {}) on main thread
DEBUG - speech.manager.SpeechManager._handleIndex (01:47:20.431) - MainThread (6624):
Unknown index 10450, speech probably cancelled from main thread.
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:21.303) - RemoteSpeechHandler_1 (3812):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 67
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:21.303) - RemoteSpeechHandler_1 (3812):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BE30> for command <SpeechCommand.CANCEL: 67>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:21.324) - MainThread (6624):
Executing <bound method SynthDriver.cancel of SynthDriver('WorldVoice')>((), {}) on main thread
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:21.964) - RemoteSpeechHandler_2 (14384):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 83
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:21.964) - RemoteSpeechHandler_2 (14384):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BD50> for command <SpeechCommand.SPEAK: 83>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:21.995) - MainThread (6624):
Executing <bound method SynthDriver.speak of SynthDriver('WorldVoice')>((['Reading requirements file in utf八 encoding   blindpandas 斜杠 bookworm at c六七八f六a   Git Hub - Google Chrome  ', IndexCommand(10451)],), {}) on main thread
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:22.283) - RemoteSpeechHandler_3 (17748):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 67
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:22.283) - RemoteSpeechHandler_3 (17748):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BE30> for command <SpeechCommand.CANCEL: 67>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:22.289) - MainThread (6624):
Executing <bound method SynthDriver.cancel of SynthDriver('WorldVoice')>((), {}) on main thread
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:22.393) - RemoteSpeechHandler_0 (21164):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 83
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:22.394) - RemoteSpeechHandler_0 (21164):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BD50> for command <SpeechCommand.SPEAK: 83>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:22.427) - MainThread (6624):
Executing <bound method SynthDriver.speak of SynthDriver('WorldVoice')>((['Copy  ', '按钮  ', IndexCommand(10454)],), {}) on main thread
DEBUG - speech.manager.SpeechManager._handleIndex (01:47:22.772) - MainThread (6624):
Unknown index 10454, speech probably cancelled from main thread.
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:23.203) - RemoteSpeechHandler_1 (3812):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 67
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:23.203) - RemoteSpeechHandler_1 (3812):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BE30> for command <SpeechCommand.CANCEL: 67>
DEBUG - external:addons.rdAccess.lib.protocol.wrapper (01:47:23.225) - MainThread (6624):
Executing <bound method SynthDriver.cancel of SynthDriver('WorldVoice')>((), {}) on main thread
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:23.429) - RemoteSpeechHandler_2 (14384):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 87
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:23.430) - RemoteSpeechHandler_2 (14384):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BDD0> for command <SpeechCommand.PLAY_WAVE_FILE: 87>
DEBUG - external:globalPlugins.rdAccess.handlers.remoteSpeechHandler._command_playWaveFile (01:47:23.430) - RemoteSpeechHandler_2 (14384):
Received PLAY_WAVE_FILE command: {'fileName': 'waves\\focusMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandlerStore.__call__ (01:47:24.045) - RemoteSpeechHandler_3 (17748):
Getting handler on <addons.rdAccess.lib.protocol.CommandHandlerStore object at 0x0EA0B530> to process command 87
DEBUG - external:addons.rdAccess.lib.protocol.CommandHandler.__call__ (01:47:24.045) - RemoteSpeechHandler_3 (17748):
Calling <addons.rdAccess.lib.protocol.CommandHandler object at 0x0DE1BDD0> for command <SpeechCommand.PLAY_WAVE_FILE: 87>
DEBUG - external:globalPlugins.rdAccess.handlers.remoteSpeechHandler._command_playWaveFile (01:47:24.046) - RemoteSpeechHandler_3 (17748):
Received PLAY_WAVE_FILE command: {'fileName': 'waves\\browseMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
LeonarddeR commented 1 year ago

Could you please execute the following code snippet on the python console for both client and server, and report back the behavior? I don't need logging this time:

d = {'fileName': 'waves\\focusMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
import nvwave
nvwave.playWaveFile(**d)
cary-rowen commented 1 year ago

Executing in the NVDA Python console on the server using the code you provided I don't find any output, nor hear a sound.

Below is the output on the client:

>>> d = {'fileName': 'waves\\focusMode.wav', 'asynchronous': True, 'isSpeechWaveFileCommand': False}
>>> import nvwave
>>> nvwave.playWaveFile(**d)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "nvwave.pyc", line 686, in playWaveFile
  File "wave.pyc", line 510, in open
  File "wave.pyc", line 160, in __init__
FileNotFoundError: [Errno 2] No such file or directory: 'waves\\focusMode.wav'
LeonarddeR commented 1 year ago

Now this is strange. Executing nvwave.playWaveFile("waves\\focusMode.wav") works on both my client and server.

Could you provide the output of the following python code on the console?

import os
print(os.getcwd())
import globalVars
print(globalVars.appDir)
cary-rowen commented 1 year ago

I'm really sorry for the trouble, I executed the code you provided above and found it was caused by another add-on(AudioManager):

>>> import os
>>> print(os.getcwd())
C:\Users\cary-rowen\AppData\Roaming\nvda\addons\audioManager\globalPlugins\audioManager
>>> import globalVars
>>> print(globalVars.appDir)
C:\Program Files (x86)\NVDA
LeonarddeR commented 1 year ago

No problem. Could you please file an issue against audio manager to request them not to change the working dir? This is really not something an add-on should do.

cary-rowen commented 1 year ago

Hi,

Could you please file an issue against audio manager to request them not to change the working dir? This is really not something an add-on should do.

Yeah, I have contacted the developer and hope that AudioManager can solve this problem soon. Thanks again.