MSVC build environment.

[kdschlosser]

using setuptools or distutils to set up an MSVC build environment has been flaky at best. It doesn't always work correctly because of Microsoft's inability to stick to a single kind of an installation ecosystem. Starting with Visual Studio 2017 Microsoft added a series of COM interfaces to Visual Studio that allows for programmatically enumerating all Visual Studio installations. This coupled with registry keys and known locations on the system drive a proper build environment is able to be set up without using any of the vcvars*.bat files. Those bat files do not always work properly and they should not be used to set up the build environment. This is what setuptools and distutils use to create the build environment and that is the reason why it does not work 100% of the time.

wxPython is now able to be built on a Windows 7 computer that is running Visual Studio 2019+ where as it was limited to Windows 10+ computers before. This was due to it using bash.exe which is apart of the WSL (Windows Subsystem for Linux) which is only available on Windows 10+. There is another way... Seeing as how most users that would want to build wxPython on a Windows computer are probably going to have Git installed why not use the bash.exe that is included with that??? So I added in detecting if Git is installed and collecting the installation path from the registry and use that bash.exe if WSL is not available.

[kdschlosser]

HEY!!! Finally.. all windows builds are passing. This should make it a bit easier when removing distutils as the mechanism to handle the compiling. Because the environment gets set any creation of new processes by the build script will cause the environment to get carried over into the new process...

The msvc script does detect cmake and ninja and I can add a property to the VisualCInfo class so the build script will be able to know if ninja is available and if it is then ninja can be used for compiling wxWidgets which would trim down the time it takes to compile. ninja and cmake are easier to install and use because they are apart of VisualStudio tho I would have to check and see if they get installed by default.

Right now the comtypes dependency is pointing to a reporitory I made that has a couple of fixes in it. When a new version of comtypes gets released that has these fixes included then we can change that dependency so it will download directly from pypi

[oleksis]

Happy to see is working. Waiting for numpy release version for Python 3.11 and see if this work. If is possible keeping notes

[kdschlosser]

I am as well. It shouldn't break at all if a newer version of Python is used. What will break is the whole wxPython build system when Python 3.12 if released because of the removal if distutils. I do not feel that distutils should be removed with a minor version change as the entire packaging system has been built around distutils for how long now? They are not giving enough time to make the migration and they don't even have a suitable replacement in place yet!!!

[swt2c]

@kdschlosser is there anything wxPython-specific about the modules in buildtools/msvc? If not, have you considered publishing those modules separately?

[oleksis]

I think python_msvc will public in PyPI and then just use like devel-requirements

[swt2c]

I think python_msvc will public in PyPI and then just use like devel-requirements

Oh, great! Yeah I think that would be preferred over bundling the code in wxPython. Thanks @kdschlosser @oleksis for your work.

[kdschlosser]

WARNING LONG READ

There is nothing that is wxPythonspecific in the build environment code. I could release it on pypi but the problems would be as in wxPython is collecting it programmatically and not installing it into a users site-packages when it is only going to get used for a single time.

setuptools offers a way to collect packages that are only used for the purposes of building a library and puts them into a temporary folder and not into a users site-packages. Unfortunatly this feature is now deprecated and there is no mechanism available to do this.

In older versions of setuptools the wheel package was not needed in order to use that feature. Now wheel is needed and that in and of it's self now makes it impossible to use without installing one time use packages into the users site-packages.

The build system in wxPython is it's own animal, it is not apart of the setup program so requirements,txt does not get parsed unless it is specifically told to. So that has to be known if compiling wxPython.

Say a user compiles wxPython using virtualenv and then wants to install wxPython into their production environment. Fantastic!! so the user runs this command to install the compiled wxPython. pip install . Now all of the packages listed in the requirements file are going to get installed into the production environment including the packages that only get used to compile wxPython.

See where the issue is??

setuptools has a mechanism to handle this and is slated for removal with no replacement because it has been slated for removal it has not been maintained and no longer functions properly.

you have pyproject.toml which is supposed to handle the build system. It is defined in PEPs 517, 518, 621 and 660. It has been around for 6 years and doesn't get used by much. I can understand as to why because this is the limit of the documentation on how to use it. https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/

which is exceedingly limited and leave you more lost on how to use it then when you initially started reading the documentation.

In PEP518 There is specific mention as to various file type parsers and which ones are apart of the standard library. It states there is no toml parser that is apart of the standard library that that "pytoml" is only 300 lines of code. It does not state if pytoml needs to be installed or not. It does not mention the inclusion of pytoml into pip or any other build system for that matter. And when visiting pytoml's repository it is soon discovered that the library has been deprecated and is no longer being developed. This leaves a large ? as to whether or not the pyproject.toml file is supported properly. The lack of information and the information that is provided leads to deprecated modules leads one to believe it is an idea that has become stale because it never gained any traction.

One of the PEPs mentions that setuptools "setup_requires" parameter uses a hook that is built into pip in order to utilize the section of the pyproject.toml file that installs build modules. After inspection of the setuptools code that handles setup_requires there is no hook into pip that is used, in fact subprocess is used to call pip to install the package into an alternate directory. It does not use any hook located in pip.

The people over at Python have not thought things through with coming up with a unified build/install ecosystem. They have jumped the gun and started to deprecate things with no replacement available or even thought of. Here is the warning that setup tools outputs when using setup_requires.

"setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer."

That leaves a person with a giant ? over their head. It provides no indication as to what the replacement is.

This is the reason why thee are 3rd party build systems that exist like poetry. They have slated distutils for removal in < 2 years. with the replacement being setuptools and setuptools has a slew of things that have been deprecated in it with no replacements available. setuptools also relies on external modules to work. To complicate things further neither setuptools nor the modules it requires are apart of the stdlib. Everything that is needed in order to build and install a package for python should be included in the stdlib and if it is something like an external compiler say MSVC then the build system needs to be able to locate those external programs. This is where a lot of the failure comes in is that the majority of the people that develop and contribute to Python are the minority OS users and they do not use Windows and have no clue about writing code to set up a proper MSVC build environment. They also do not accept PRs that have been made to correct problems in Windows and bug reports pertaining to Windows go largely unanswered/ignored/unfixed.

Now being that in one of the PEPs regarding pyproject.toml and it stating that setuptools uses a hook to facilitate package installs using the mechanisms put into place in pip and setuptools having a deprecated warning about that mechanism it leads one to believe that pyproject.toml files are on the out and are not going to be supported, If that is not the case then why would it be deprecated in setuptools?

So now do we install packages that are only needed to compile wxPython and not have them get installed into the user normal Python environment when they are not needed to run wxPython???

If someone can answer that question for me then I will entertain the idea of releasing it to pypi. If there is no way then the code needs to be added directly to a packages build system which would make it a moot point to release it on pypi. To top it off I am having to battle with comtypes not being maintained like it should be and it not building properly due to it's reliance on build_py2to3 which is apart of distutils prior to python 3.9. This component was never deprecated and it was removed with zero notice.

[oleksis]

The last job pipeline don't run nmake. Compare the output from the CI pipeline

Pipeline working OK ✅

##[section]Starting: build wxWidgets
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.195.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
[command]"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
[command]"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/4c7790eb-a9a2-4bab-b040-41f24396f18f.sh
which: no cl.exe in (/mingw64/bin:/usr/bin:/c/Users/VssAdministrator/bin:/c/Users/VssAdministrator/AppData/Roaming/Python/Python39/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86:/c/agents/2.196.2/externals/git/cmd:/c/agents/2.196.2/externals/git/mingw64/bin:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.3/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.1/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.2/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.15.15/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.312-7/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.4/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/cmd:/mingw64/bin:/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/VssAdministrator/.dotnet/tools:/c/Users/VssAdministrator/.cargo/bin:/c/Users/VssAdministrator/AppData/Local/Microsoft/WindowsApps)
Will build using: "C:\hostedtoolcache\windows\Python\3.9.9\x86\python.exe"
3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 17:50:04) [MSC v.1929 32 bit (Intel)]
Python's architecture is 32bit
cfg.VERSION: 4.1.2a1

Running command: build_wx
Machine architecture: x64
Build architecture: x86

== Python =====================================================
  version: 3.9.9.final.0
  architecture: x86
  library: Python39.lib
  libs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\libs']
  includes: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\cpython', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\internal']

== Visual Studio Enterprise 2019==============================================
   description:        Scalable, end-to-end solution for teams of any size
   version:            16.11.32106.194
   version (friendly): 2019
   display version:    16.11.9
   path:               C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise
   executable:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe
   is complete:        True
   is prerelease:      False
   is launchable:      True
   state:              local, registered, no reboot required

== Visual C ===================================================
   version: 14.29.30139
   path:    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC

   -- Tools ---------------------------------------------------
      version:     14.29.30133
      path:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133
      redist path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\
   -- F# ------------------------------------------------------
      path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools
   -- DLL -----------------------------------------------------
      version: v142-14.29.30139.0
      path:    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\x86\Microsoft.VC142.CRT

== MSBuild ====================================================
   version: 16.11.2.50704
   path:    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe

== HTML Help ==================================================
   path: C:\Program Files (x86)\HTML Help Workshop

== ATLMFC =====================================================
   path:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC
   include path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include
   lib path:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x86

== Windows SDK ================================================
   version:     10.0
   sdk version: 10.0.22000.0
   path:        C:\Program Files (x86)\Windows Kits\10\

== Universal CRT ==============================================
   version:           10.0.22000.0
   path:              C:\Program Files (x86)\Windows Kits\10\
   lib directory:     C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22000.0\ucrt\x86\
   headers directory: C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt\

== Extension SDK ==============================================
   path: C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs

== TypeScript =================================================
   path: C:\Program Files (x86)\Microsoft SDKs\TypeScript\4.3

== .NET =======================================================
   version:    v4.7.3062.2

   -- x86 -----------------------------------------------------
      version: v4.7.3062.2
      path:    C:\Windows\Microsoft.NET\Framework
   -- x64 -----------------------------------------------------
      version: v4.7.3062.2
      path:    
   -- NETFX ---------------------------------------------------
      path:         None
      x86 exe path: None
      x64 exe path: None

Downloading microsoft.web.webview2 1.0.622.22...
Connection successful...
Data downloaded...
wxWidgets build options: ['--wxpython', '--unicode', '--jobs=4']
Updating wx/msw/setup.h
setting build options...
nmake.exe -f makefile.vc UNICODE=1 OFFICIAL_BUILD=1 COMPILER_VERSION=140 SHARED=1 MONOLITHIC=0 USE_OPENGL=1 USE_GDIPLUS=1 BUILD=release

Microsoft (R) Program Maintenance Utility Version 14.29.30139.0
Copyright (C) Microsoft Corporation.  All rights reserved.

    if not exist vc140_mswudll mkdir vc140_mswudll
    if not exist ..\..\lib\vc140_dll mkdir ..\..\lib\vc140_dll
    if not exist ..\..\lib\vc140_dll\mswu mkdir ..\..\lib\vc140_dll\mswu
    if not exist ..\..\lib\vc140_dll\mswu\wx mkdir ..\..\lib\vc140_dll\mswu\wx
    if not exist ..\..\lib\vc140_dll\mswu\wx\setup.h copy ..\..\include\wx\msw\setup.h ..\..\lib\vc140_dll\mswu\wx\setup.h
        1 file(s) copied.

Pipeline failed ❌

##[section]Starting: build wxWidgets
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.195.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
[command]"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
[command]"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/480453ed-f1f1-424b-b8de-96364d8549c5.sh
which: no cl.exe in (/mingw64/bin:/usr/bin:/c/Users/VssAdministrator/bin:/c/Users/VssAdministrator/AppData/Roaming/Python/Python39/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86:/c/agents/2.196.2/externals/git/cmd:/c/agents/2.196.2/externals/git/mingw64/bin:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.3/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.1/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.2/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.15.15/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.312-7/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.4/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/cmd:/mingw64/bin:/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/VssAdministrator/.dotnet/tools:/c/Users/VssAdministrator/.cargo/bin:/c/Users/VssAdministrator/AppData/Local/Microsoft/WindowsApps)
Will build using: "C:\hostedtoolcache\windows\Python\3.9.9\x86\python.exe"
3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 17:50:04) [MSC v.1929 32 bit (Intel)]
Python's architecture is 32bit
cfg.VERSION: 4.1.2a1

Running command: build_wx
Machine architecture: x64
Build architecture: x86

== Python =====================================================
  version: 3.9.9.final.0
  architecture: x86
  library: Python39.lib
  libs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\libs']
  includes: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\cpython', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\internal']

== Visual Studio Enterprise 2019 ==============================
   description:        Scalable, end-to-end solution for teams of any size
   install date:       Sun Jan 16 22:12:32 2022
   version:            16.11.32106.194
   version (friendly): 2019
   display version:    16.11.9
   path:               C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise
   executable:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe
   is complete:        True
   is prerelease:      False
   is launchable:      True
   state:              local, registered, no reboot required

== Visual C ===================================================
   version:   14.29.30139
   path:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC
   has cmake: True
   has ninja: True

   -- Tools ---------------------------------------------------
      version:     14.29.30133
      path:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133
      redist path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\
   -- F# ------------------------------------------------------
      path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools
   -- DLL -----------------------------------------------------
      version: v142-14.29.30139.0
      path:    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\x86\Microsoft.VC142.CRT

== MSBuild ====================================================
   version: 16.11.2.50704
   path:    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe

== HTML Help ==================================================
   path: C:\Program Files (x86)\HTML Help Workshop

== ATLMFC =====================================================
   path:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC
   include path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include
   lib path:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x86

== Windows SDK ================================================
   version:     10.0
   sdk version: 10.0.22000.0
   path:        C:\Program Files (x86)\Windows Kits\10\

== Universal CRT ==============================================
   version:           10.0.22000.0
   path:              C:\Program Files (x86)\Windows Kits\10\
   lib directory:     C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22000.0\ucrt\x86\
   headers directory: C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt\

== Extension SDK ==============================================
   path: C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs

== TypeScript =================================================
   path: C:\Program Files (x86)\Microsoft SDKs\TypeScript\4.3

== .NET =======================================================
   version:    v4.7.3062.2

   -- x86 -----------------------------------------------------
      version: v4.7.3062.2
      path:    C:\Windows\Microsoft.NET\Framework
   -- x64 -----------------------------------------------------
      version: v4.7.3062.2
      path:    
   -- NETFX ---------------------------------------------------
      path:         None
      x86 exe path: None
      x64 exe path: None

Downloading microsoft.web.webview2 1.0.622.22...
Connection successful...
Data downloaded...
wxWidgets build options: ['--wxpython', '--unicode', '--jobs=4']
Updating wx/msw/setup.h
setting build options...
Command 'cl.exe' failed with exit code 1.
Finished command: build_wx (0m2.523s)
##[error]Bash exited with code '1'.
##[section]Finishing: build wxWidgets

[kdschlosser]

The build environments are identical so I am not sure why it didn't compile properly. I will set the distutils logging to capture the full output to see what is going on.

[kdschlosser]

found the problem and fixed it.

[oleksis]

Now is the order in the PATH for find the properly executables for example:

Git link.exe versus MSVC link.exe

[kdschlosser]

Heh didn't think about that being an issue. Gonna have to make sure the old path statements are at the end of the line.

The last commit hopefully will hopefully trim down the time it takes to compile on Windows without using JOM.

[kdschlosser]

It took about 15 minutes to compile on my antique AMD FX 6 core @ 3.2ghz.

[kdschlosser]

OK path is now fixed.

I had to do a little tinkering with it because of the subprocess calls that the build system uses and it running the msvc script multiple times. I didn't want to end up with multiple entries for the same thing like the path.

[kdschlosser]

2 minutes 54 seconds to compile wxWidgets on Azure. Don't know what it was previously but that seems to be pretty quick.

[oleksis]

Just need pass the wxPython build with waf

Check this result ok versus result failed

[kdschlosser]

I think I got it fixed. we will see. what happens ¯\_(ツ)_\/¯

[oleksis]

2 minutes 54 seconds to compile wxWidgets on Azure. Don't know what it was previously but that seems to be pretty quick.

This is the last CI success. For example the Windows CI Py37_x64 Job have Duration: 41m 36s

[kdschlosser]

it was taking about 10 minutes to compile wxWidgets and if I can get the changes I made to fully work it will trim that tome down to 3 minutes making the total build time about 34 minutes or so.

[kdschlosser]

ok so for whatever reason waf is ignoring the INCLUDE environment variable. So I have to figure out how to pass it as a command line argument. Not sure what the nmake build does because I cannot find that path anywhere in the nmake files.

[oleksis]

The setup.h is located in many places like "..\wxPython\Phoenix\ext\wxWidgets\include\msvc\wx\setup.h" or in build wxWidgets

[kdschlosser]

I found where the issue is. Hopefully I have it fixed now, we will see...

[oleksis]

I found where the issue is. Hopefully I have it fixed now, we will see...

Permission denied in the generated file

[oleksis]

waflib uses its procedures to detect and use 'msvc' (vcvarsall.bat): "..\wxPython\Phoenix\bin\waf3-2.0.22-1241519b19b496207abef1f72bbf61c2\waflib\Tools\msvc.py"

[oleksis]

waflib uses its procedures to detect and use 'msvc' (vcvarsall.bat): "..\wxPython\Phoenix\bin\waf3-2.0.22-1241519b19b496207abef1f72bbf61c2\waflib\Tools\msvc.py"

Here the source in gitlab

[oleksis]

I think we need translate the macros and commands from nmake to cmake

With nmake the project compile in type "BUILD=release", so using cmake we need "-DCMAKE_BUILD_TYPE=Release"

then "wxbase31u.lib" is different like "wxbase315ud_vc_custom.dll"

[oleksis]

This is the "build.cfg" with nmake

WXVER_RELEASE=5 
BUILD=release 
MONOLITHIC=0 
SHARED=1 
UNICODE=1 
TOOLKIT=MSW 
TOOLKIT_VERSION= 
WXUNIV=0 
CFG= 
VENDOR=custom 
OFFICIAL_BUILD=1 
DEBUG_FLAG=1 
DEBUG_INFO=1 
RUNTIME_LIBS=dynamic 
USE_EXCEPTIONS=1 
USE_RTTI=1 
USE_THREADS=1 
USE_AUI=1 
USE_GUI=1 
USE_HTML=1 
USE_MEDIA=1 
USE_OPENGL=1 
USE_QA=1 
USE_PROPGRID=1 
USE_RIBBON=1 
USE_RICHTEXT=1 
USE_STC=1 
USE_WEBVIEW=1 
USE_XRC=1 
COMPILER=vc 
COMPILER_VERSION=140 
CC=cl 
CXX=cl 
CFLAGS= 
CPPFLAGS= 
CXXFLAGS= 
LDFLAGS= 

[oleksis]

Here some changes for execute the CI using MSVC build envirinment using CMake and Ninja

• build wxWidgets from 7m 26s to 3m 19s
• Changes to the files buildtools/build_wxwidgets.py , buildtools/config.py
• CI for wxPython Phoenix on Azure DevOps (only Windows)
• For cmake not include compiler version in wxCOMPILER_PREFIX
• build.py is the patch for bash in WSL2 and the fix MSW DLL dir for cmake
• TODO: Reproducible builds

[oleksis]

Hi @kdschlosser

Here the msvc_compiler_cmake_ninja branch if want apply the commits on your fork

[kdschlosser]

I did also discover the differences with the macros. I had not gone through the nmake files and the cmake file to make a translation between the 2.

I have been looking at several aspects of the build system as a whole and what can be done to speed it up across the board for all OS's.

I wrote a build system a while back for a different project I was working on and I never ended up using it. I am going to remove it from the other project and make a repository for it. It might take me a couple of days to do that. I want you to take a look at it as it might have some value.

[oleksis]

Ok as soon as the repository is ready, share the link to take a look.

[kdschlosser]

OK I will.

I took a look at the changes you made and I added them a test version I am tinkering with. There is a problem with the changes you made in that it breaks a windows compilation when ninja is not available. That is because of the change to the VCDLL in config.

I did however find a different way to solve the problem. and that's to set the cmake macro wxBUILD_CUSTOM_SETUP_HEADER_PATH to point to one directory below where the setup.h is located.

I actually corrected all of the macros so the proper cmake macros get set.

I do have to dink about getting the compiled dll names correct and have to finish adding the code to rename the vc_x64_dll folder that cmake creates to vc140_x64_dll which is what it should be. On the off chance somewhere else in the setup program or the build it uses that name directly.

The foldername has to remain the same because of nmake. the creation of that folder resides in the nmake makefiles for wxWidgets and unless it gets changed to a constant so it doesn't use the compiler version when making it we need to make sure it stays that way.

[oleksis]

Any update, when is ready let me know, and I test it. The best case is build wxPython using any build system and become Reproducible builds

[kdschlosser]

Here is one coming down the pipe for ya.

I added cleaning the ninja and cmake files. I had also found a place that was messing with the build environment. the distutils msvc compiler was being started for the sole purpose of getting the link and compiler flags but the distutils msvc compiler monkeys around with the build environment when it is initialized. That's probably why all of my efforts before to add the included setup.h file failed. LOL would figure!!!

[kdschlosser]

Oh, one other thing I noticed since I did finally manage to get to the end of the build, waf does compile using multiple cores so doing anything with the build system I made would just make a lot of work with no speed increase. Tho we can still look at things like speeding up the cmd_etg function or we could launch the build for wxWidgets in a thread making sure to leave a single core available and do the etg code generation while wxWidgets is compiling.

That would shave some time off the build I would think.

[oleksis]

I think renaming the Dlls broken the finals tests

[oleksis]

@kdschlosser

This is the Fix to the VC dll dir and Dont rename vc custom libraries

[kdschlosser]

It's going to be a few days before I will get back to this. My computer has been having a technical issue where it just turns of out of no where. No BSOD, no errors.. nothing that I know of is overheating. All fans are working. It just powers off completely and I have top unplug it for a second and plug it back in and it will turn back on. Completely random when it does it.

So last night it did it and I lost a bunch of code I has just written. That was the last straw, so today I went and picked up a new computer. AMD Threadripper pro 3955wx - 16 cores @ 3.9/4.3 ghz, 64gb RAM, 2 m.2 SSD's and 4 2tb HDDs. I have to migrate everything over to the new machine and that is going to take some time. It has Windows 10 Pro already installed so I am going to use that. I haven't used windows 10 much at all so I am not familiar with using it. It is going to take me a while to get everything moved over and all of my applications installed onto the new machine.

It's a hell of a machine. 6 PCIe x16 4.0 slots, 8 hot swap RAID bays, 4 m.2 Nvme PCIe 4.0 ports, 2TB max RAM and a 64 core processor will work in it.

My old machine is close to 10 years old so I really can't bitch about it at all. it has been on 24/7 for almost the entire time, I reinstalled the OS only a single time and that is when I put an SSD in it.

[oleksis]

How glad I am that you can use a new and better PC. As soon as you have an update you notify me to go testing. I wish you a better experience and happy hacking code!

[oleksis]

@kdschlosser

Here you can see the HTML report output.html with diffoscope

Comparation build system nmake vs (cmake and ninja)

First directory with nmake vs Second directory with (cmake and ninja)

$ diffoscope --html output.html --exclude '__pycache__' venv/Lib/site-packages/wx/ ./wx/

[oleksis]

Comparation build system nmake vs (cmake and ninja)

• Same last source code

  With nmake (No ninja : "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.ORIG.exe")

  $ diffoscope --html output.html --exclude '__pycache__' ./wx/ ./wx-cmake-ninja/

output.html

»       DLL·Name:·wxbase315u_vc140_x64.dll	65	»       DLL·Name:·wxbase315u_vc_custom.dll

[RobinD42]

@kdschlosser, thanks for this PR. I've been fighting a little with trying to rely more on setuptools and kept running into issues. Then I remembered that you had already worked on this and it looks like exactly what we need. I'll be working on understanding it a bit better and maybe experimenting or tinkering a bit, and then I'll try to get it committed in the next day or two.

[swt2c]

@kdschlosser, thanks for this PR. I've been fighting a little with trying to rely more on setuptools and kept running into issues. Then I remembered that you had already worked on this and it looks like exactly what we need. I'll be working on understanding it a bit better and maybe experimenting or tinkering a bit, and then I'll try to get it committed in the next day or two.

My concern with this is that it seems like an awful lot of code (6000 lines!) that would now have to be maintained in wxPython. Aren't there other Python projects that run into trouble locating MSVC? Seems like this would be a common problem that others would have run into.

[RobinD42]

Yes, that's a good point. There's some refactoring that could be done to reduce and simplify somewhat. Also I wonder if vswhere.py is doing the same or similar as MS's vswhere.exe tool and, if so, could we fetch info from that instead?

I could almost get what I need from setuptools, but I keep running into this bug no matter how I install newer versions of MSVC: https://github.com/pypa/setuptools/issues/1902. It's been around for a few years and the fix seems simple so I'm not sure why they haven't done it already. If I could get my build modifications working reliably with setuptools that would be my preference. Maybe I should just monkey-patch it...

[kdschlosser]

You can vswhere.exe that being said you are calling an executable using subprocess which can have issues in and of itself like an orphaned process if the program crashes for some reason. The other things is vswhere doesn't expose the entire COM API where as the vswhere script I wrote does. vswhere.exe uses the exact same COM API to collect information that the vswhere script does. I have also organized the information collected so it can be accessed in a more user friendly manner.

There is definitely cleanup that can be done. The MSVC script I wrote works and it works for every version of the MSVC compiler that has been used to compile Python since version 2.6. locates the C compiler for all versions in between as well. All windows SDKs that are supported by those compilers is also located.

The MSVC script has a built in backup so if for some reason the COM API doesn't work or is not available because of permissions or any other reason the script will still be able to locate what is needed. It is a slower then using the COM API but it will still work.

[kdschlosser]

I have opened up an issue on setup tools about updating the compiler detection on Windows using the same mechanisms I am. It has fallen on deaf ears like any other issue with Python running on Windows does. The developers only seem to care about it working 100% on a Posix system. I don't understand why the greater market share has the least amount of importance,

[kdschlosser]

Don't commit this as it is far from finished an will introduce other problems I am sure.

The last thing I was messing with was using Ninja to speed up the time it takes to compile on Windows.

[kdschlosser]

Phoenix has an extremely complex build system and I am pretty sure that if you and I put our heads together a far simpler build system can be written.

[marekr]

Phoenix has an extremely complex build system and I am pretty sure that if you and I put our heads together a far simpler build system can be written.

I'll just drop this breadcrumb here. I'm a KiCad core developer and we use wxPython, I'm not a fan but I needed true compatibility with our Windows CI tooling and build pipeline and python tooling is grossly incompatible for that.

The entire build chain can be simplified to a combination of python for some of the preprocessing like SIP and then the build work under CMake that can parallelize builds and be done in seconds.

This is the "preprocess" portion https://gitlab.com/kicad/packaging/kicad-vcpkg-registry/-/blob/main/ports/wxpython/portfile.cmake

And here is the CMakeList for the pyd compilation and packaging https://gitlab.com/kicad/packaging/kicad-vcpkg-registry/-/blob/main/ports/wxpython/CMakeLists.txt

As you notice, it's actually a vcpkg port. However, upstream microsoft vcpkg isn't meant for python plugins so it rests in our kicad registry instead

[kdschlosser]

I use KiCad for my PCB designs. I thought you guys use QT for the GUI framework.

You can use Python to compile your entire program. I never check to see if KiCAD is cross platform or not. If it is it can still be compiled using python. I have a framework that I wrote that does a really good job of locating compilers on different operating systems.

I did know KiCAD has Python scripting support but never investigated how much KiCAD relied on Python. I am really curious, probably because I use KiCAD. If you want to have a more in depth conversation about build systems or if you want to know anything about this MSVC script I wrote let me know. Tell me where to go to have have the conversation and I would be more then happy to.