Use build of wxPython which has symbols

[jcsteh]

Spun off #5998.

We're seeing quite a few crashes in wx code, but we can't get any info from them because wxPython doesn't include symbols. We want to use a build of wxPython for which we have symbols.

Possible approaches:

1. wxWidgets do provide pre-built dlls, libs and symbols. I tried just replacing the wxPython dlls with these, but wxPython refused to load them. Still, this is worth checking again.
2. Even if that fails, perhaps there's some way to make wxPython's build system use the wxWidgets libs/dlls. This would at least avoid the need to build wxWidgets.
3. Failing all of that, we'll have to build wxPython from scratch ourselves, including wxWidgets.

CC @feerrenrut, @MichaelDCurran.

[feerrenrut]

Double checked wxWidgets version required is 3.0.2.0 (by looking at the file properties in miscDeps\python\wx\wxmsw30u_stc_vc90.dll

wxWidgets downloads: https://github.com/wxWidgets/wxWidgets/releases/tag/v3.0.2 Downloaded: wxMSW-3.0.2_vc90_ReleaseDLL.7z

Compared the checksums with the dlls in our wxPython libs directory. They do not match. Dropped in the dlls from the wxWidgest release and got the following error:

INFO - __main__ (07:09:38):
Starting NVDA
CRITICAL - __main__ (07:09:38):
core failure
Traceback (most recent call last):
  File "nvda.pyw", line 189, in <module>
    import core
  File "core.py", line 23, in <module>
    import nvwave
  File "nvwave.py", line 14, in <module>
    import wx
  File "C:\work\nvda\miscDeps\python\wx\__init__.py", line 45, in <module>
    from wx._core import *
  File "C:\work\nvda\miscDeps\python\wx\_core.py", line 16924, in <module>
    from _gdi import *
  File "C:\work\nvda\miscDeps\python\wx\_gdi.py", line 4, in <module>
    import _gdi_
ImportError: DLL load failed: The specified procedure could not be found.

[feerrenrut]

It looks like we would have to build wxWidgets included with wxPython to get the appropriate dlls / pdbs.

See: https://wxpython.org/builddoc.php.

[feerrenrut]

However I tried downloading the 32-bit binaries for MSVC 9 from https://wxpython.org/download.php This is an installer, which includes pre-built version of wxWidgets and pdb's. The checksums for all these files matches the files we currently have in NVDA. When dropped into nvda there does not seem to be a problem.

[jcsteh]

Oh! I didn't realise those downloads included pdbs. Very nice. I think that means all we need do is grab the symbols and upload them to our symbol server; no changes to NVDA necessary. That also means we can analyse existing crash dumps rather than waiting for those crashes to be captured again, which is even better.

[michaelDCurran]

Accept for pdbs for the pyds. I.e. this is just WX symbols, not wxPython symbols. To be completely covered we'd still want to build wxPython with symbols if possible. But I think 99% of what we need already is covered by the existing wx symbols as the pyds are only wrapper functions generated by swig.

On 5/07/2016 7:59 PM, James Teh wrote:

Oh! I didn't realise those downloads included pdbs. Very nice. I think that means all we need do is grab the symbols and upload them to our symbol server; no changes to NVDA necessary. That also means we can analyse existing crash dumps rather than waiting for those crashes to be captured again, which is even better.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/6143#issuecomment-230437258, or mute the thread https://github.com/notifications/unsubscribe/ANf5nYPTuGl7y1ymDZz5mCUQwcQsMNdBks5qSisOgaJpZM4JEzIy.

Michael Curran Executive Director, NV Access Limited Phone: +61 7 3149 3306 Website: http://www.nvaccess.org/ Twitter: @NVAccess Facebook: http://www.facebook.com/NVAccess

[jcsteh]

It's probably worth analysing one of the current crash dumps where we crash in wx with these symbols present. If that works, I'd lean towards not building wxPython ourselves, as it's more maintenance work for us going forward.

[michaelDCurran]

These symbols do work. Using them on one crash dump allows kp and friends to list more than 5 frames. It also moves the "stack information not available. Following frames may be wrong" message down to above the most inner call to a wxPython pyd function. (it used to be above the most inner wx dll function). However, from that point onwards, things are all over the place. A lot of 0x0s and it gives up after stack 21.

So annoyingly as this is, for perfect results the wxPython pdbs will still help.

But, in most cases, hopefully the inner most frames will show the issue.

Most of these though are the deactivateActCtx bug.

Mick

On 5/07/2016 8:22 PM, James Teh wrote:

It's probably worth analysing one of the current crash dumps where we crash in wx with these symbols present. If that works, I'd lean towards not building wxPython ourselves, as it's more maintenance work for us going forward.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/6143#issuecomment-230441838, or mute the thread https://github.com/notifications/unsubscribe/ANf5nbwya4ObajL0QJ4010GJg45okShHks5qSjBggaJpZM4JEzIy.

Michael Curran Executive Director, NV Access Limited Phone: +61 7 3149 3306 Website: http://www.nvaccess.org/ Twitter: @NVAccess Facebook: http://www.facebook.com/NVAccess

[michaelDCurran]

In short, we would still like to proceed with investigating whether it is possible to generate symbols for wxPython pyd files, as this will ensure we have all symbols allowing for a full and accurate stack when viewing NVDA in a debugger.

[feerrenrut]

Building wxPython on Windows 10 Instructions:

• Make sure that you have Python 2.7 32 bit installed and in your PATH
• Download Microsoft Visual C++ Compiler for Python 2.7 and Install
  • You will need to search for this. It should be available from Microsoft.
  • Use Visual C++ 2008 32-bit Command Prompt that is added to your system for all compilation steps
• Install .net Framework 3.5
  • You will need to search for this. It should be available from Microsoft.
• Make a build directory eg C:\wxBuild
• Clone wxPython source from: git@github.com:wxWidgets/wxPython.git into your wxBuild directory
  • Checkout the tag for the release we want: git checkout wxPy-3.0.2.0
  • You should now have C:\wxBuild\wxPython
• Get wxWidgets pre-built:
  • Download wxWidgets3.0-devel-win32-3.0.2.0.exe
  • See the 32-bit binaries for MSVC 9 link on https://wxpython.org/download.php
  • Install to a path like C:\wxWidgets\3.0.2.0-win32
  • Copy from C:\wxWidgets\3.0.2.0-win32\* to C:\wxBuild\wxWidgets
  • Copy C:\wxBuild\wxWidgets\lib\vc90_dll to C:\wxBuild\wxPython\lib\vc90_dll
• Ensure that we build pdb files for wxPython:
  • Edit C:\wxBuild\wxPython\wxPython\config.py
  • at around line 998:
  • add cflags += ['/Zi']
  • add lflags = ['/DEBUG' ]
  • Edit C:\BuildWx\wxPython\wxPython\build-wxpython.py
  • at around line 263 (after if statement to check for windows):
  • add DESTDIR = options.installdir
  • add PREFIX = options.prefix
• Get some other dependencies:
  • Start a terminal that has access to bash
  • navigate to C:\wxBuild\wxPython\wxPython
  • run bin/subrepos-make
  • This will checkout some other dependencies that are required.

Start the build

• Open the visual c++ 2008 command prompt
• Navigate to C:\wxBuild\wxPython\wxPython
• Set the following environment variables:
  • SET DISTUTILS_USE_SDK=1
  • SET MSSdk=1
• Remember to set these again each time you restart the command prompt.
• run python build-wxpython.py --no_wxbuild --install > build.log
• You could then tail build.log to see progress.

Check the results

• The outputs of the build should be in C:\BuildWx\wxPython\wxPython\build\lib.win32-2.7\wx
  • There will also be outputs in C:\wxBuild\wxPython\wxPython\wx and <your python install>\Lib\site-packages but we want the one in ...build\lib.win32-2.7\wx since this matches best with what is currently in nvda
• Compare checksums for the wxWidgets dlls with their source (in C:\wxBuild\wxWidgets\lib\vc90_dll)
• Check that there are pdb files for the pyd files

Finish up

• Copy dll and pdb files from wxWidgets source (C:\wxBuild\wxWidgets\lib\vc90_dll) into the output.

Errors?

• error: Unable to find vcvarsall.bat
  • look at http://stackoverflow.com/questions/2667069/cannot-find-vcvarsall-bat-when-running-a-python-script
  • Try updating your python setup tools
  • pip install -U setuptools
• c:\work\bldwxpy3\wxPython\include\wx\platform.h(183) : fatal error C1083: Cannot open include file: 'wx/setup.h': No such file or directory
  • Remember to copy the wxWidgets\lib\vc90_dll to wxPython (see above)
• error: package directory 'wx\lib\agw' does not exist
  • Remember to run bin/subrepos-make from the right directory (see above)

[michaelDCurran]

Fixed in 8f252398f4bf121009fd4b205c729629eab22352