search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.53k stars 1.19k forks source link

[FR] Autodetect Visual Studio 2022 #3933

Open levicki opened 1 year ago

levicki commented 1 year ago

What's the problem this feature will solve?

Current setuptools\msvc.py says:

Known supported compilers:
--------------------------
Microsoft Visual C++ 14.X:
    Microsoft Visual C++ Build Tools 2015 (x86, x64, arm)
    Microsoft Visual Studio Build Tools 2017 (x86, x64, arm, arm64)
    Microsoft Visual Studio Build Tools 2019 (x86, x64, arm, arm64)

This may also support compilers shipped with compatible Visual Studio versions.

I see no good reason why Visual Studio 2022 is not among them? It could use the same mechanism as 2019 for detection.

Describe the solution you'd like

Not having to manually invoke vcvarsall.bat and do set DISTUTILS_USE_SDK=1 or edit other people's setup scripts to do that would be nice.

Alternative Solutions

No response

Additional context

Many AI projects need building some bits here and there (think torch extensions and the like), and right now they cannot build with only Visual Studio 2022 installed because it does not seem to be detected.

Code of Conduct

abravalheri commented 1 year ago

Hi @levicki would you consider providing a PR adding this functionality to either setuptools (or also likely https://github.com/pypa/distutils)?

levicki commented 1 year ago

@abravalheri Of course I wouldn't mind helping, but I am not very proficient with Python myself and at the moment I have no understanding on how this system even works. If you don't mind providing some pointers perhaps I could help?

jllllll commented 1 year ago

@abravalheri Of course I wouldn't mind helping, but I am not very proficient with Python myself and at the moment I have no understanding on how this system even works. If you don't mind providing some pointers perhaps I could help?

I just took a look through setuptools and distutils MSVC code and saw that both use vswhere.exe to locate the latest Visual Studio with MSVC installed. The key here is that it looks for vswhere.exe in the default install location under \Program Files (x86)\Microsoft Visual Studio\Installer. Apparently, no matter where you install VS to, the installer will always place vswhere.exe under that location. https://github.com/microsoft/vswhere

Check to see if it is there. If it is, there may be a more fundamental issue in how Visual Studio is getting located.

Relevant lines: https://github.com/pypa/setuptools/blob/b8c0a599676a23048a9041233b423c9dfaf4b70a/setuptools/msvc.py#L82-L95 https://github.com/pypa/distutils/blob/4435cec31b8eb5712aa8bf993bea3f07051c24d8/distutils/_msvccompiler.py#L78-L95

abravalheri commented 1 year ago

Thank you very much for the analysis @jllllll, indeed it seems like the approach using vswhere should be compatible with Visual Studio 2022...

@levicki, I am also not very knowledgeable myself on how setuptools deals with visual studio (the code was mostly contributed by other maintainers/volunteers).

As pointed out by @jllllll, the code that deals with the compiler information comes mainly from distutils._msvccompiler.

Setuptools augments distutils._msvccompiler to deal with edge cases using setuptools.msvc. If I understood correctly, the main setuptools "upgrade" is to replace distutils._msvccompiler._get_vc_env with setuptools.msvc.msvc14_get_vc_env.

levicki commented 1 year ago

@jllllll, @abravalheri

Check to see if it is there. If it is, there may be a more fundamental issue in how Visual Studio is getting located.

Yes, it is there and if I manually execute:

vswhere.exe -latest -prerelease -requiresAny -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -requires Microsoft.VisualStudio.Workload.WDExpress -property installationPath -products *

I get the response:

C:\Program Files\Microsoft Visual Studio\2022\Professional

No idea why it is not detected from the script though. Any ideas how I could construct a minimal test case? Like import setuptools or distutils and run some command to see what I get?

EDIT: Not sure if what I tried makes sense:

C:\>pip list setuptools
Package    Version
---------- -------
pip        23.1.2
setuptools 65.5.0

C:\>python
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.msvc._msvc14_find_vc2015()
(14, 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\')
>>> setuptools.msvc._msvc14_find_vc2017()
(15, 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Auxiliary\\Build')
>>> setuptools.msvc._msvc14_find_vcvarsall("amd64")
('C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Auxiliary\\Build\\vcvarsall.bat', 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Auxiliary\\Build\\..\\..\\redist\\MSVC\\14.36.32532\\spectre\\onecore\\x64\\Microsoft.VC143.CRT\\vcruntime140.dll')

I have no idea why it is picking Spectre mitigated and OneCore flavor of the runtime DLL — I don't think that either of those should be a default choice.

levicki commented 1 year ago

@abravalheri Maybe this is not a problem of setuptools, but a problem with pytorch cpp_extension way of (mis?)using it. I am trying to figure it out, but my basic level knowledge of Python is not helping. If it turns out that setuptools is not to blame I will update this ticket.

abravalheri commented 1 year ago

Hi @levicki thank you very much for investigating this. What I would do is exactly what you are trying (i.e., manually running the functions of setuptools.msvc and distutils._msvccompiler in a terminal)...

I don't know if this helps but note that when building a package the version of setuptools used might differ from the one you have installed in your system (both pypa/build and pypa/pip will, by default, create a new temporary virtual environment with the versions specified in pyproject.toml's [build.system] requires). Maybe there is an older version being used?

levicki commented 1 year ago

@abravalheri Maybe I am not understanding the question but to me it seems that both the test environment I used, and the one where I had trouble building pytorch extension originally due to auto-detection not working have the same version of setuptools installed (the one shown above).

abravalheri commented 1 year ago

Hi @levicki, are you using build or pip to build your package?

If you using just python -m build or pip install/wheel, they will create a separated virtual environment to build your project. In those environments a new version of setuptools will be installed according to the version specified in pyproject.toml.

If you want to ensure that the version of setuptools that you have installed in your system is the one being used you need to python -m build --no-isolation or pip install/wheel --no-build-isolation. Note however that you will need to install all the build dependencies yourself, e.g. wheel, Cython, etc...

levicki commented 1 year ago

@abravalheri Actually it was python setup.py install (which I am aware is deprecated).

The weird thing is that when I tried building it again now, it is building without me changing anything and without having to manually run vcvarsall.bat.

The only error in the log now comes from torch\cpp_extension.py:

C:\PROGRAMS\OOBABOOGA\venv\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')

This is one of those headscratching moments...

Perhaps the best outcome of this issue should be to update documentation to say that VS 2022 is already supported?

abravalheri commented 1 year ago

If anyone can successfully confirm that it works for VS 2022, it should be fine to update the comment at the top of the msvc.py file (I haven't used it myself, but I am happy to review a PR for other members of the community that might have).

Is there other part of the setuptools docs that also mention this?

jllllll commented 1 year ago

I never had the issue of VS 2022 not being detected. However, I see others reporting it all the time, only for the issue to resolve itself eventually for unknown reasons. Whatever vswhere.exe uses to locate VS must have some delay before taking effect.

abravalheri commented 1 year ago

Hi @jllllll, thank you very much for the information. I can see in the vswhere issue tracker some comments about this:

So it seems that by default vswhere will not include "incomplete" installations:

> vswhere.exe -h
Visual Studio Locator version 3.1.4+c7a417bfb4 [query version 3.0.4492.23473]
Copyright (C) Microsoft Corporation. All rights reserved.

Usage:  [options]

Options:
  -all           Finds instances in complete, launchable, and incomplete states. By default, only instances
                 in a complete state - no errors or reboot required - are searched.
  -prerelease    Also searches prereleases. By default, only releases are searched.
...

I don't know how vswhere classifies "incomplete states", but would it be a possibility that those users you mention found their installation in a transient "incomplete state"? This would explain why vswhere does not find them... (however the help text seem to suggest that a reboot could solve the "incomplete state").

jllllll commented 1 year ago

@abravalheri One thing to consider that I had forgotten about until now is the Windows "feature" of "Fast Startup" (may have other names not sure). Essentially, it saves the state of memory to disk before powering off so that the pc can skip the boot process and start faster. I wonder if this is why restarts don't seem to fix the issue for people? To my knowledge, this feature is enabled by default in Windows 10+. I know that I disabled this on my system to prevent issues like this from occurring.

levicki commented 1 year ago

@abravalheri My installation was definitely not in an incomplete state so I doubt that was the cause.

@jllllll I also have fast startup disabled so it's not that either.

Perhaps there is some problem with environment variables? Mine is a bit specific because I have quite a long PATH and all entries in it living in Program Files or Program Files (x86) are written as %ProgramFiles% and %ProgramFiles(x86)%. Maybe the environment block gets too long and chopped off once all is expanded and then starting stuff fails?

I am out of ideas really.