search

nvaccess / nvda

NVDA, the free and open source Screen Reader for Microsoft Windows
https://www.nvaccess.org/
Other
2.1k stars 634 forks source link

Application with multibyte file name raises error #9583

Closed nishimotz closed 5 years ago

nishimotz commented 5 years ago

Steps to reproduce:

https://www.dropbox.com/s/66xdm39bis2x03k/test_app.zip?dl=0

"テストアプリケーション.exe" is provided by @riku22 in https://github.com/nvdajp/nvdajp/issues/198

Actual behavior:

Expected behavior:

System configuration

NVDA installed/portable/running from source:

NVDA version:

alpha snapshots nvda_snapshot_alpha-17134,363af94b.exe and later

Windows version:

Name and version of other software in use when reproducing the issue:

Other information about your system:

Other questions

Does the issue still occur after restarting your PC?

Have you tried any other versions of NVDA? If so, please report their behaviors.

The issue does not occur with alpha snapshots until nvda_snapshot_alpha-17104,6d4b1121.exe

the log is as follows:

INFO - __main__ (23:39:35.487):
Starting NVDA
INFO - core.main (23:39:36.844):
Config dir: C:\Users\nishimotz\AppData\Roaming\nvda
INFO - config.ConfigManager._loadConfig (23:39:36.844):
Loading config: C:\Users\nishimotz\AppData\Roaming\nvda\nvda.ini
INFO - core.main (23:39:37.005):
Developer Scratchpad mode enabled
INFO - core.main (23:39:38.299):
NVDA version alpha-17134,363af94b
INFO - core.main (23:39:38.301):
Using Windows version 10.0.17763 workstation
INFO - core.main (23:39:38.301):
Using Python version 2.7.16 (v2.7.16:413a49145e, Mar  4 2019, 01:30:55) [MSC v.1500 32 bit (Intel)]
INFO - core.main (23:39:38.301):
Using comtypes version 1.1.3
INFO - core.main (23:39:38.301):
Using configobj version 5.1.0 with validate version 1.0.1
INFO - synthDriverHandler.setSynth (23:39:40.190):
Loaded synthDriver oneCore
INFO - core.main (23:39:40.190):
Using wx version 4.0.3 msw (phoenix) wxWidgets 3.0.5
INFO - brailleInput.initialize (23:39:40.194):
Braille input initialized
INFO - braille.initialize (23:39:40.196):
Using liblouis version 3.9.0
INFO - braille.BrailleHandler.setDisplayByName (23:39:40.198):
Loaded braille display driver noBraille, current display has 0 cells.
INFO - _UIAHandler.UIAHandler.MTAThreadFunc (23:39:40.684):
UIAutomation: IUIAutomation6
INFO - core.main (23:39:42.381):
NVDA initialized
ERROR - core.CorePump.run (23:40:14.450):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - queueHandler.flushQueue (23:40:14.525):
Error in func update from eventQueue
Traceback (most recent call last):
  File "queueHandler.pyc", line 53, in flushQueue
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.562):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 883, in pumpAll
  File "IAccessibleHandler.pyc", line 651, in processFocusWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.658):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.713):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.755):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.825):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:17.686):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
DrSooom commented 5 years ago

CC: @leonardder

LeonarddeR commented 5 years ago

@nishimotz: Could you please provide appropriate logs? You might also be able to provide a pr for this? Note that fixing to Python 3 might fix this without any effort.

nishimotz commented 5 years ago

@leonardder the log is contained in the issue above.

I did not identify how to fix it so far.

nishimotz commented 5 years ago

I am not sure it works, but I have created the PR as #9590

nishimotz commented 5 years ago

It seems that AppVeyor CI is failing, because pygetwindow, which is used by pyautogui, is broken.

nishimotz commented 5 years ago

Tested with nvda_snapshot_pr9590-17270,ad4d45bd.exe. The issue is fixed with PR #9590. Please review the PR.

LeonarddeR commented 5 years ago

@leonardder the log is contained in the issue above.

You mean the Japanese issue right? Please provide a direct reference to the log in this issue (i.e. directly copy the traceback here or provide an URL to a log file).

nishimotz commented 5 years ago

I have tested with the Japanese Shift-JIS (code page 932) environment.

When the focus moves to the app, error raises as follows.

ERROR - core.CorePump.run (23:40:14.450):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - queueHandler.flushQueue (23:40:14.525):
Error in func update from eventQueue
Traceback (most recent call last):
  File "queueHandler.pyc", line 53, in flushQueue
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.562):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 883, in pumpAll
  File "IAccessibleHandler.pyc", line 651, in processFocusWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.658):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.713):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.755):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:14.825):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
ERROR - core.CorePump.run (23:40:17.686):
errors in this core pump cycle
Traceback (most recent call last):
  File "core.pyc", line 489, in run
  File "IAccessibleHandler.pyc", line 897, in pumpAll
  File "IAccessibleHandler.pyc", line 614, in processGenericWinEvent
  File "appModuleHandler.pyc", line 137, in update
  File "appModuleHandler.pyc", line 123, in getAppModuleFromProcessID
  File "appModuleHandler.pyc", line 174, in fetchAppModule
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)
LeonarddeR commented 5 years ago

Thanks for providing the log.

I can reproduce this by simply adding an é to the filename of a random executable. I think this is caused by NVDA trying to encode a bytes object (which is already encoded) to mbcs. IN that case, Python first decodes the string and than re-encodes it. It seems that decoding the string fails here.

TO reproduce, just try this:

"é".encode("mbcs")

nishimotz commented 5 years ago

Related issue is reporeted as follows:

ERROR - scriptHandler.executeScript (21:30:36.365):
error executing script: <bound method GlobalCommands.script_reportAppModuleInfo of <globalCommands.GlobalCommands object at 0x04E2B690>> with gesture u'NVDA+\u30b3\u30f3\u30c8\u30ed\u30fc\u30eb+f1'
Traceback (most recent call last):
  File "scriptHandler.pyc", line 192, in executeScript
  File "globalCommands.pyc", line 1631, in script_reportAppModuleInfo
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 0: ordinal not in range(128)

Now I understand that the issue is caused by PR #8727.

appModuleHandler.py:

def getAppNameFromProcessID(processID,includeExt=False):
    """Finds out the application name of the given process.
    @param processID: the ID of the process handle of the application you wish to get the name of.
    @type processID: int
    @param includeExt: C{True} to include the extension of the application's executable filename, C{False} to exclude it.
    @type window: bool
    @returns: application name
    @rtype: unicode or str
    """
    if processID==NVDAProcessID:
        return "nvda.exe" if includeExt else "nvda"
    FSnapshotHandle = winKernel.kernel32.CreateToolhelp32Snapshot (2,0)
    FProcessEntry32 = processEntry32W()
    FProcessEntry32.dwSize = ctypes.sizeof(processEntry32W)
    ContinueLoop = winKernel.kernel32.Process32FirstW(FSnapshotHandle, ctypes.byref(FProcessEntry32))
    appName = unicode()

(snipped)

    # This might be an executable which hosts multiple apps.
    # Try querying the app module for the name of the app being hosted.
    # Python 2.x can't properly handle unicode module names, so convert them.
    # No longer the case in Python 3.
    if sys.version_info.major == 2:
        appName = appName.encode("mbcs")

I cannot understand why appName allows both unicode (for Python 3) and mbcs (for Python 2.7).

IMO, importlib.import_module() should be monkey-patched to take unicode if Python 2.7 is used, rather than what I did in PR #9590.

nishimotz commented 5 years ago

Sorry, I have corrected the last two sentences in the comment above. I welcome the discussion.

nishimotz commented 5 years ago

Created new PR as #9610. PR #9590 is a more conservative work around this, but need to be squashed. I hope either of them should be considered for 2019.2.

LeonarddeR commented 5 years ago

Ugh, I wasn't aware that this is a regression. That means we should treat this a bit diffrently.

Commit 6d4b1121 only seems to bump the version. I wonder whether this could be the update from Python 2.7.15 to 2.7.16.

nishimotz commented 5 years ago

Created PR #9635 against the beta branch.

josephsl commented 5 years ago

Hi,

I think rather than reverting the possibly offending code (part of which is app module handler code), it would be best to perform version checks as a temporary workaround with a source code comment reminding @feerrenrut about this problem. Note that in Python 3, decoding a Unicode string with MBCS results in creation of a bytes object, something we should avoid if possible except in special cases (I myself ran into this problem while researching #7105).

Thanks.

LeonarddeR commented 5 years ago

@josephsl: Are there user noticeable improvements in #8727? If not, there's no reason to keep it in master/beta if it breaks things. I'm afraid that merging #9635 introduces more risk than reverting #8727.

Cc @feerrenrut

josephsl commented 5 years ago

Hi,

There is no user-level improvements in #8727, however importlib is the way to go for import internals in Python 3. I'll prepare a reversion PR just in case reverting should take highest of highest of priorities.

Thanks.

LeonarddeR commented 5 years ago

@josephsl commented on 31 May 2019, 08:59 CEST:

... importlib is the way to go for import internals in Python 3.

That's true. But if importlib creates issues for Python 2 that require workarounds, I think we should defer using importlib to threshold.

LeonarddeR commented 5 years ago

This should be fixed as of #9657. Please request a reopen if otherwise.