pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.52k stars 1.19k forks source link

[BUG] DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. #3985

Open lxrswdd opened 1 year ago

lxrswdd commented 1 year ago

setuptools version

setuptools=68.0.0

Python version

Python 3.10.11

OS

Windows 11

Additional environment information

No response

Description

It keeps saying that despite the fact that I have installed MSVC v143 buildtool and Windows 11 SDK.

DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

Expected behavior

I am trying to run a code that was written based on StyleGan2 https://github.com/NVlabs/stylegan3/blob/main/training/networks_stylegan2.py

How to Reproduce

It is really hard to reproduce given the codes involved in multiple codes. Unless download the full package.

Output

The full output of the error can be found here.

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File [e:\Anaconda\envs\face_landmark\lib\site-packages\setuptools\msvc.py:173](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/setuptools/msvc.py:173), in _msvc14_get_vc_env(plat_spec)
    172 try:
--> 173     out = subprocess.check_output(
    174         'cmd /u /c "{}" {} && set'.format(vcvarsall, plat_spec),
    175         stderr=subprocess.STDOUT,
    176     ).decode('utf-16le', errors='replace')
    177 except subprocess.CalledProcessError as exc:

File [e:\Anaconda\envs\face_landmark\lib\subprocess.py:421](file:///E:/Anaconda/envs/face_landmark/lib/subprocess.py:421), in check_output(timeout, *popenargs, **kwargs)
    419     kwargs['input'] = empty
--> 421 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    422            **kwargs).stdout

File [e:\Anaconda\envs\face_landmark\lib\subprocess.py:526](file:///E:/Anaconda/envs/face_landmark/lib/subprocess.py:526), in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command 'cmd /u /c "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 && set' returned non-zero exit status 255.

The above exception was the direct cause of the following exception:

DistutilsPlatformError                    Traceback (most recent call last)
Cell In[2], line 20
     16 with torch.no_grad():
     18     x = 0; y = 0; z = 0 # z is not used
---> 20     img_rendered, _, _ = render_face(0, yaw, 0, x, y, z, code_info)
     22     # plot
     23     plt.subplot(1,7,i+1)

Cell In[1], line 112, in render_face(pitch, yaw, roll, x, y, z, code)
    108 cam_info = gen_cam(pitch, yaw, roll)
    110 cam_info["batch_Tvecs"][0,2,0] -= 0.5
--> 112 ret = decoder(mode='test', batch_xy=batch_xy, batch_uv=batch_uv, code=code, update_emas=False, **cam_info)
    113 fmap = ret['coarse_dict']['merge_fmap'].permute(0,2,3,1).detach().cpu().numpy()
    114 depth = ret['coarse_dict']['depth'].permute(0,2,3,1).detach().cpu().numpy()

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\nn\modules\module.py:1501](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\./modules\StyleMorpheus.py:356, in StyleMorpheusNet.forward(self, mode, batch_xy, batch_uv, code, update_emas, batch_Rmats, batch_Tvecs, batch_inv_inmats, dist_expr, **kwargs)
    351 """
    352 code: enhanced 3dmm codes
    353 """
    355 # mapping: 3dmm space --> style space
--> 356 style_code = self.mapping_nets(code, update_emas)
    358 shape_code = torch.cat([style_code['id'],  style_code['exp']], 1)
    359 appea_code = torch.cat([style_code['tex'], style_code['ill']], 1)

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\nn\modules\module.py:1501](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\./modules\StyleMorpheus.py:69, in DisentangledMappingNet.forward(self, z_codes, update_emas)
     67 w_codes = {}
     68 for key in self.code_dims:
---> 69     w_codes[key] = self.mapping_nets[key](z=z_codes[key], c=None, update_emas=update_emas)[:,0,:]
     70 return w_codes

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\nn\modules\module.py:1501](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\modules\stylegan\training\networks_stylegan2.py:246, in MappingNetwork.forward(self, z, c, truncation_psi, truncation_cutoff, update_emas)
    244 for idx in range(self.num_layers):
    245     layer = getattr(self, f'fc{idx}')
--> 246     x = layer(x)
    248 # Update moving average of W.
    249 if update_emas and self.w_avg_beta is not None:

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\nn\modules\module.py:1501](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\modules\stylegan\training\networks_stylegan2.py:124, in FullyConnectedLayer.forward(self, x)
    122 else:
    123     x = x.matmul(w.t())
--> 124     x = bias_act.bias_act(x, b, act=self.activation)
    125 return x

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\./modules/stylegan\torch_utils\ops\bias_act.py:84, in bias_act(x, b, dim, act, alpha, gain, clamp, impl)
     82 assert isinstance(x, torch.Tensor)
     83 assert impl in ['ref', 'cuda']
---> 84 if impl == 'cuda' and x.device.type == 'cuda' and _init():
     85     return _bias_act_cuda(dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp).apply(x, b)
     86 return _bias_act_ref(x=x, b=b, dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp)

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\./modules/stylegan\torch_utils\ops\bias_act.py:41, in _init()
     39 global _plugin
     40 if _plugin is None:
---> 41     _plugin = custom_ops.get_plugin(
     42         module_name='bias_act_plugin',
     43         sources=['bias_act.cpp', 'bias_act.cu'],
     44         headers=['bias_act.h'],
     45         source_dir=os.path.dirname(__file__),
     46         extra_cuda_cflags=['--use_fast_math', '--allow-unsupported-compiler'],
     47     )
     48 return True

File [d:\A\OneDrive](file:///D:/A/OneDrive) - UBC\Projects\3D_HEAD\Experiment\methods\style5.0\./modules/stylegan\torch_utils\custom_ops.py:136, in get_plugin(module_name, sources, headers, source_dir, **build_kwargs)
    134     # Compile.
    135     cached_sources = [os.path.join(cached_build_dir, os.path.basename(fname)) for fname in sources]
--> 136     torch.utils.cpp_extension.load(name=module_name, build_directory=cached_build_dir,
    137         verbose=verbose_build, sources=cached_sources, **build_kwargs)
    138 else:
    139     torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\utils\cpp_extension.py:1284](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/utils/cpp_extension.py:1284), in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1192 def load(name,
   1193          sources: Union[str, List[str]],
   1194          extra_cflags=None,
   (...)
   1202          is_standalone=False,
   1203          keep_intermediates=True):
   1204     r'''
   1205     Loads a PyTorch C++ extension just-in-time (JIT).
   1206 
   (...)
   1282         ...     verbose=True)
   1283     '''
-> 1284     return _jit_compile(
   1285         name,
   1286         [sources] if isinstance(sources, str) else sources,
   1287         extra_cflags,
   1288         extra_cuda_cflags,
   1289         extra_ldflags,
   1290         extra_include_paths,
   1291         build_directory or _get_build_directory(name, verbose),
   1292         verbose,
   1293         with_cuda,
   1294         is_python_module,
   1295         is_standalone,
   1296         keep_intermediates=keep_intermediates)

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\utils\cpp_extension.py:1509](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/utils/cpp_extension.py:1509), in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1505                 hipified_sources.add(hipify_result[s_abs]["hipified_path"] if s_abs in hipify_result else s_abs)
   1507             sources = list(hipified_sources)
-> 1509         _write_ninja_file_and_build_library(
   1510             name=name,
   1511             sources=sources,
   1512             extra_cflags=extra_cflags or [],
   1513             extra_cuda_cflags=extra_cuda_cflags or [],
   1514             extra_ldflags=extra_ldflags or [],
   1515             extra_include_paths=extra_include_paths or [],
   1516             build_directory=build_directory,
   1517             verbose=verbose,
   1518             with_cuda=with_cuda,
   1519             is_standalone=is_standalone)
   1520 finally:
   1521     baton.release()

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\utils\cpp_extension.py:1624](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/utils/cpp_extension.py:1624), in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
   1622 if verbose:
   1623     print(f'Building extension module {name}...', file=sys.stderr)
-> 1624 _run_ninja_build(
   1625     build_directory,
   1626     verbose,
   1627     error_prefix=f"Error building extension '{name}'")

File [e:\Anaconda\envs\face_landmark\lib\site-packages\torch\utils\cpp_extension.py:1870](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/torch/utils/cpp_extension.py:1870), in _run_ninja_build(build_directory, verbose, error_prefix)
   1867 plat_name = distutils.util.get_platform()
   1868 plat_spec = PLAT_TO_VCVARS[plat_name]
-> 1870 vc_env = distutils._msvccompiler._get_vc_env(plat_spec)
   1871 vc_env = {k.upper(): v for k, v in vc_env.items()}
   1872 for k, v in env.items():

File [e:\Anaconda\envs\face_landmark\lib\site-packages\setuptools\msvc.py:214](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/setuptools/msvc.py:214), in msvc14_get_vc_env(plat_spec)
    212 # Always use backport from CPython 3.8
    213 try:
--> 214     return _msvc14_get_vc_env(plat_spec)
    215 except distutils.errors.DistutilsPlatformError as exc:
    216     _augment_exception(exc, 14.0)

File [e:\Anaconda\envs\face_landmark\lib\site-packages\setuptools\msvc.py:178](file:///E:/Anaconda/envs/face_landmark/lib/site-packages/setuptools/msvc.py:178), in _msvc14_get_vc_env(plat_spec)
    173     out = subprocess.check_output(
    174         'cmd /u /c "{}" {} && set'.format(vcvarsall, plat_spec),
    175         stderr=subprocess.STDOUT,
    176     ).decode('utf-16le', errors='replace')
    177 except subprocess.CalledProcessError as exc:
--> 178     raise distutils.errors.DistutilsPlatformError(
    179         "Error executing {}".format(exc.cmd)
    180     ) from exc
    182 env = {
    183     key.lower(): value
    184     for key, _, value in
    185     (line.partition('=') for line in out.splitlines())
    186     if key and value
    187 }
    189 if vcruntime:

DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
abravalheri commented 1 year ago

Hi @lxrswdd, could you please run in the same machine you are trying to build the package, the following command in a classic windows terminal (cmd, not powershell) and paste the output here so we can have a look?

cmd /u /c "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 && set
lxrswdd commented 1 year ago

Thank you @abravalheri

I just ranthem in cmd

image

I also try them in an Anaconda env. It suggests the same thing \Microsoft was unexpected at this time.

abravalheri commented 1 year ago

That is weird and very difficult to debug without having direct access to the machine 😅. Sorry, I will have to ask you a few more things before being able to help, if that is OK:

lxrswdd commented 1 year ago
  1. I do have C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat I also wish to mention that I have C:\Program Files (x86)\Microsoft Visual Studio\2019 but I kind of installed all the buildtools required for both 2019 and 2022.
  2. Sorry, I don't quite get what you meant here. I run the python in an Anaconda virtual env.
  3. Please see the attachments image image

@abravalheri

abravalheri commented 1 year ago

2. Sorry, I don't quite get what you meant here. I run the python in an Anaconda virtual env.

Environment variables are a "feature" of the operating system, not directly related to the Python/Anaconda concept of virtual envs (although virtual envs may add/change environment variables when active).

Since in your other comments the folder C:\Program Files (x86) exists and seem to be the home of vswhere and vcvarsall, I am assuming that the value for the ProgramFiles(x86) environment variable in your system is C:\Program Files (x86).

Regarding the attachments you sent, it is OK that the command fails for PowerShell, the syntax I gave you is only valid in the classic windows cmd[^1].


So... it seems that there is a problem with your installation of vcvarsall.bat (maybe it got corrupted somehow?)[^2]. Running the following line in a classic windows cmd prompt should result in a list of environment variables:

cmd /u /c "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 && set

We can break down this line into 2 parts:

I am not sure why the first part is not working. Have you tried uninstalling it and installing it again?

[^1]: You probably need to prepend the command with & to make it work on PowerShell (as in & 'C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe' -latest -prerelease -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath -products *) [^2]: Setuptools will always use the latest version you got installed (this is something hardcoded and not something that we can change for now)

abravalheri commented 1 year ago

It might also be a matter of weird encoding happening on cmd... Any chance it is breaking when you try to point out the file path C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat which contains some weird chars (e.g. ` or(`). In theory this should be fine, specially considering that we have been using quotes...

lxrswdd commented 1 year ago
  1. Now I have uninstalled 2019 and 2022 using the visual studio installer. Then I installed the 2022 community with a checklist below. I opened cmd as administrator. However, the problem persists when I run "C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 && set .
image
  1. Then I ran logging again and got something below.
image

I also attached the installation list.

Screenshot 2023-07-18 132023
abravalheri commented 1 year ago

I can see that even if you cd into the C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build directory and run vcvarsall.bat x86_amd64 you still get the failure, which rules out the possibility of cmd messing with the command encoding, and therefore the conclusion is that there is something inside the vcvarsall.bat script that is not compatible with your system/environment.

I found some people reporting this problem in other contexts:

Please have a look on those. At a first glance, it seems that there is a chance your PATH environment variable is misconfigured (e.g., invalid chars), which breaks vcvarsall.

lxrswdd commented 1 year ago

That getting weird, It appears that my PATH environment variable is well configured. There is not any quotation marks.

C:\Users\FunkYou>echo %PATH%
E:\Cuda\bin;E:\Cuda\libnvvp;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;E:\软件\Bandizip\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.2.0\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;E:\Matlab\bin;C:\Program Files\Pandoc\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;D:\软件\;C:\ProgramData\chocolatey\bin;C:\Program Files\RedHat\java-1.8.0-openjdk-1.8.0.372-1\bin;C:\Program Files\RedHat\java-1.8.0-openjdk-1.8.0.372-1\jre\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\FunkYou\AppData\Local\Programs\Eclipse Adoptium\jdk-17.0.7.7-hotspot\bin;E:\Anaconda;E:\Anaconda\Library\mingw-w64\bin;E:\Anaconda\Library\usr\bin;E:\Anaconda\Library\bin;E:\Anaconda\Scripts;C:\Users\FunkYou\AppData\Local\Microsoft\WindowsApps;E:\软件\Microsoft VS Code\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\cl.exe;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx86\x86;

My user variables

image

My system variables

image
abravalheri commented 1 year ago

I also am not sure what is happening, but I think we manage to track down the root of the problem (from setuptools perspective).

Setuptools needs to run "C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 to be able to use Microsoft Visual C++. This process is described by Microsoft's docs at https://learn.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-170. However that command seems to be failing in your machine, which is making setuptools think that the installation is not valid (maybe setuptools' error message is not very user friendly and needs improvement, but I believe this is the most likely explanation).

The next best step for you now might be to seek advice from Microsoft on why vcvarsall.bat x86_amd64 is failing and how to fix it[^2]. Once vcvarsall.bat x86_amd64 works, please try to install/run the code in StyleGan2 and let us know if the problem with setuptools still persists[^1].

[^1]: Also if you find a solution to make vcvarsall.bat x86_amd64 work, could you please share it here? It might be beneficial in the future for other devs.

[^2]: It seems that you can set the VSCMD_DEBUG environment variable to 3 to get more information what is going on on vcvarsall.bat x86_amd64, maybe that can help to find more details.

GGuuu commented 6 months ago

I finally solved this problem❗

Following this link, this error occurs when quotation mark(") exists in your PATH!

When I run echo %PATH% on cmd, I found quotation mark in PATH list. ex) C:\Users;"C:\Users{user}"; <-- That ' " ' mark is problem

After remove and rewrite path in system variation, this error had been solved

I hope this solution helps you.