search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.5k stars 1.19k forks source link

cannot open input file '.obj' in build on distutils from Python 3.9 #2417

Closed hoffmang9 closed 3 years ago

hoffmang9 commented 4 years ago

We are currently building binary wheels for Windows under Python 3.7 and 3.8. When attempting to add Python 3.9 to the mix - in both cibuildwheel and directly in a Windows python 3.9 venv - quotes are being omitted around the link call.

I originally reported it in cibuildwheel here - https://github.com/joerick/cibuildwheel/issues/441

Quoting that here: I'm attempting to build wheels on python 3.7, 3.8, and now 3.9.

With recent changes in setuptools a couple of weeks ago I was forced to add: CIBW_ENVIRONMENT_WINDOWS: SETUPTOOLS_USE_DISTUTILS=stdlib

To have the Windows MSVC build for 3.8 correctly put quotes around LIBPATH entries in the link command line. This entry is showing it improperly unquoted: /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\ATLMFC\lib\x64

Without the quotes it's then trying to link from C:\Program Files\ instead of the actual lib directories.

My new attempt to build with python 3.9 gave me this new error: C:\cibw\python\python.3.9.0-rc1\tools\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first. at - https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true#step:13:1697

The unquoted Windows link attempt is here: https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true#step:13:1883

Overall run can be found here - https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true

jaraco commented 4 years ago

I'm surprised to see distutils_patch involved if you have the latest version of Setuptools or if you have SETUPTOOLS_USE_DISTUTILS=stdlib.

I'm not sure I know what steps I'd need to take to replicate the error. Are you able to replicate the error locally? Can you derive the steps I would need to take on a clean Windows box to replicate the issue?

hoffmang9 commented 4 years ago

I'm using that work around to not have the same problem building wheels for 3.7 and 3.8.

All you need is Windows, python 3.9 and msvc community edition. Make a venv and then pip wheel .

hoffmang9 commented 4 years ago

I could also set up a branch so you can use our ci instead.

jaraco commented 3 years ago

I tried running pip wheel . on a checkout of Chia-Network/chiapos, but I'm not even sure if that's the same code where you have the issue. The build did fail, but I'm not sure if it failed with the same errors as in your CI.

I'll struggle to disentangle the project-specific behaviors in chiapos from the behaviors that trigger this issue.

Can you distill the problem down to a more minimal, essential example?

hoffmang9 commented 3 years ago

My team member @wjblanke made you a super stripped down version of chiapos that only pybinds to one C++ file.

https://github.com/Chia-Network/chiapos/tree/culled-setuptools-win-issue

You can see it complete building a wheel successfully on Windows with Python 3.8 and the environment variable SETUPTOOLS_USE_DISTUTILS=stdlib set here: https://github.com/Chia-Network/chiapos/runs/1340027779?check_suite_focus=true#step:9:190

But then the exact same thing fails for Python 3.9 here: https://github.com/Chia-Network/chiapos/runs/1340027779?check_suite_focus=true#step:9:279 because there are no quotes around the link command line and thus all the spaces in Window's default directory names blow things up.

You should be able to confirm that this branch will compile on python 3.8 with SETUPTOOLS_USE_DISTUTILS=stdlib as an environment variable, but when you switch to python 3.9 the quotes will disappear and the compile will fail to link.

jaraco commented 3 years ago

Awesome. Thanks for putting those together. I've confirmed the issue as you've described. Quotes are missing on Python 3.9 and on older Pythons with SETUPTOOLS_USE_DISTUTILS=local. I believe now that points to:

jaraco commented 3 years ago

I've traced the regression to python/cpython#18743.

jaraco commented 3 years ago

I'm not sure it's an issue with spaces in the filenames. That change does change the way the command is rendered in the logs, but I'm confident that Popen passes each command-line parameter discretely. However, I have seen situations where the underlying process (link.exe in this case) relies on the parameter having literal quotes.

jaraco commented 3 years ago

The error I see is cannot open input file '.obj'

PS C:\Users\jaraco\local\chiapos> py setup.py build
WARNING: The wheel package is not available.
running build
running build_ext
building 'chiapos' extension
"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe" /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\jaraco\local\chiapos\.eggs\pybind11-2.6.0-py3.9.egg\pybind11\include -Ic:\users\jaraco\local\chiapos\.eggs\pybind11-2.6.0-py3.9.egg\pybind11\include -Isrc -I. -IC:\Python39\include -IC:\Python39\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tppython-bindings/chiapos.cpp /Fobuild\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /EHsc /std:c++17 /O2 /DVERSION_INFO=\\\"0.0.0\\\"
chiapos.cpp
C:\Users\jaraco\local\chiapos\python-bindings\../src/plotter_disk.hpp(36): warning C4005: '_HAS_STD_BYTE': macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include\yvals_core.h(554): note: see previous definition of '_HAS_STD_BYTE'
"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_chiapos build\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /OUT:build\lib.win-amd64-3.9\chiapos.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib ""
LINK : fatal error LNK1181: cannot open input file '.obj'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\link.exe' failed with exit code 1181

But when I run that exact same command, I get a different error:

PS C:\Users\jaraco\local\chiapos> & "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_chiapos build\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /OUT:build\lib.win-amd64-3.9\chiapos.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib ""
   Creating library build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib and object build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.exp
Generating code
Finished generating code
LINK : fatal error LNK1158: cannot run 'rc.exe'

I suspect the different error is because of environment variables... or possibly the context of prior commands.

jaraco commented 3 years ago

If I configure the path to match, then run the link command manually, it works:

PS C:\Users\jaraco\local\chiapos> $env:PATH='C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x86;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x86;C:\Program Files (x86)\Windows Kits\10\bin\x86;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\\MSBuild\Current\Bin;C:\Windows\Microsoft.NET\Framework\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\;C:\Python27\;C:\Python27\Scripts;C:\Python35\Scripts\;C:\Python35\;C:\Python39\Scripts\;C:\Python39\;C:\Program Files\Python39\Scripts\;C:\Program Files\Python39\;C:\Python38\Scripts\;C:\Python38\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\ProgramData\chocolatey\bin;C:\Program Files\dotnet\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Users\jaraco\AppData\Local\Microsoft\WindowsApps;C:\Users\jaraco\AppData\Local\hyper\app-3.0.2\resources\bin;C:\Users\jaraco\.local\bin;C:\Users\jaraco\AppData\Roaming\Python\Python39\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja'
PS C:\Users\jaraco\local\chiapos> & "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_chiapos build\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /OUT:build\lib.win-amd64-3.9\chiapos.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib ""
   Creating library build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib and object build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.exp
Generating code
Finished generating code
jaraco commented 3 years ago

The issue appears to be that the empty string at the end of the command:

PS C:\Users\jaraco\local\chiapos> py setup.py build
WARNING: The wheel package is not available.
running build
running build_ext
building 'chiapos' extension
"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe" /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\jaraco\local\chiapos\.eggs\pybind11-2.6.0-py3.9.egg\pybind11\include -Ic:\users\jaraco\local\chiapos\.eggs\pybind11-2.6.0-py3.9.egg\pybind11\include -Isrc -I. -IC:\Python39\include -IC:\Python39\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tppython-bindings/chiapos.cpp /Fobuild\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /EHsc /std:c++17 /O2 /DVERSION_INFO=\\\"0.0.0\\\"
> c:\users\jaraco\p\pypa\distutils\distutils\spawn.py(75)spawn()
-> proc = subprocess.Popen(cmd, env=env)
(Pdb) del cmd[-1]
(Pdb) c
chiapos.cpp
C:\Users\jaraco\local\chiapos\python-bindings\../src/plotter_disk.hpp(36): warning C4005: '_HAS_STD_BYTE': macro redefinition
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include\yvals_core.h(554): note: see previous definition of '_HAS_STD_BYTE'
"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_chiapos build\temp.win-amd64-3.9\Release\python-bindings/chiapos.obj /OUT:build\lib.win-amd64-3.9\chiapos.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib ""
> c:\users\jaraco\p\pypa\distutils\distutils\spawn.py(75)spawn()
-> proc = subprocess.Popen(cmd, env=env)
(Pdb) cmd
['C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\link.exe', '/nologo', '/INCREMENTAL:NO', '/LTCG', '/DLL', '/MANIFEST:EMBED,ID=2', '/MANIFESTUAC:NO', '/LIBPATH:C:\\Python39\\libs', '/LIBPATH:C:\\Python39\\PCbuild\\amd64', '/LIBPATH:C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\lib\\x64', '/LIBPATH:C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.18362.0\\ucrt\\x64', '/LIBPATH:C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64', '/EXPORT:PyInit_chiapos', 'build\\temp.win-amd64-3.9\\Release\\python-bindings/chiapos.obj', '/OUT:build\\lib.win-amd64-3.9\\chiapos.cp39-win_amd64.pyd', '/IMPLIB:build\\temp.win-amd64-3.9\\Release\\python-bindings\\chiapos.cp39-win_amd64.lib', '']
(Pdb) del cmd[-1]
(Pdb) c
   Creating library build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.lib and object build\temp.win-amd64-3.9\Release\python-bindings\chiapos.cp39-win_amd64.exp
Generating code
Finished generating code
jaraco commented 3 years ago

It seems that extra_link_args is set to [''] and that empty string has a different effect on subprocess.Popen than it did on spawn.

jaraco commented 3 years ago

In a8f9d49, I've patched the culled repro and the error no longer occurs. Correct me if I'm wrong, but I suspect the use of [""] was intended to mean "no args" but in fact it means "one blank arg", and now that subprocess.Popen is correctly passing that empty arg to the subprocess, the linker is trying to link a file called ./.obj, which doesn't exist. Given that the issue was unintended, incorrect usage, I doubt we'll want to invest time and energy restoring the prior, less-correct behavior in distutils... thought it might make sense for the msvccompiler to error on or remove blank command args if that can be shown to never be valid.

The rendering of the command-line turned out to be a red herring. It's true that the distutils change did also affect the rendering of the command-line, but that was incidental and generally didn't affect the behavior. I'll make a patch to the distutls code to use list2cmdline when rendering the command line for logging to make it more correct.

I'm going to close this as invalid, but please don't hesitate to comment if making this change in chiapos doesn't fix the issue.

hoffmang9 commented 3 years ago

Thanks for the assist! That's working solidly and I'll push it out to our other repos as well. We totally agree that bug for bug compatibility isn't a good idea...