Windows installations crashing due to unavailability of standard output

david-cortes commented 5 years ago

I’m not entirely sure what’s the problem here, but I have this package in PyPI: https://pypi.org/project/nonnegcg/

Whenever I try to install it Windows machines through pip or setuptools (e.g. pip install nonnegcg), I get an error at the moment something tries to write to standard output (which is bound to happen during installation), bet it print , warning, or something else, which crashes the installation:

ValueError: underlying buffer has been detached
Exception ignored in: <_io.TextIOWrapper mode='w' encoding='cp1252'>
ValueError: underlying buffer has been detached

Installing it with distutils from the downloaded file does not result in any problems (python setup.py install, modifying setup.py to import setup from distutils rather than from setuptoos), and it install just fine from pip and setup.py + setuptools in linux, but not on windows.

anntzer commented 2 years ago

I can repro a similar problem with the following minimal setup: pyproject.toml:

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "mod"
version = "42"

with an empty mod.py next to it.

Running pip wheel -v . or pip install -ve . for example succeeds, but prints out large amounts of tracebacks like

  --- Logging error ---
  Traceback (most recent call last):
    File "C:\ProgramData\Miniconda3\envs\myenv\lib\logging\__init__.py", line 1103, in emit
      stream.write(msg + self.terminator)
  ValueError: underlying buffer has been detached
  Call stack:
<elided>
  Message: 'removing build\\bdist.win-amd64\\wheel'
  Arguments: ()

abravalheri commented 2 years ago

Hi @anntzer, I am afraid I cannot reproduce the error in the Windows machine I have available (I am using Windows 10, PowerShell 5.1.19041.1682, Python 3.9.13):

mkdir test-stdout-detached
cd test-stdout-detached
# Write the given pyproject.toml example to a file using UTF-8/unix line ending encodings
echo "" > mod.py
python -m pip wheel .
# ...
# Successfully built mod

# (file mod-42-py3-none-any.whl is created successfully)

This is probably related to a specific setup you have in your environment.

I recommend trying again using an isolated virtual environment (python -m venv, not the conda ones).

anntzer commented 2 years ago

I have repro'd again this with a clean, entirely new, python.org python 3.11 install in Windows 10 Pro 21H2, whether outside of any venv or inside a clean, entirely new venv, whether in cmd.exe or in PowerShell 7.3.

abravalheri commented 2 years ago

I have repro'd again this with a clean, entirely new, python.org python 3.11 install in Windows 10 Pro 21H2, whether outside of any venv or inside a clean, entirely new venv, whether in cmd.exe or in PowerShell 7.3.

I still cannot reproduce it I am afraid 😅 Could you please share a windows docker container with the reproducer?

anntzer commented 2 years ago

Sorry, setting up a Windows docker container is a bit beyond my expertise...

abravalheri commented 2 years ago

No problems @anntzer, thank you anyway for having attempted to propose a reproducer.

I wrote a test using Dockerfile (see bellow). Everything goes well and I cannot observe the faulty behaviour described in this issue.

To keep investigating, it would be necessary that a member of the community that is experiencing this problem provide a minimal reproducer that can be executed in an isolated environment (ideally via a windows container, as shown below).

If there is no reproducer available, I think it would be safe to say that this problem is more likely to be related on how an specific environment is setup (e.g. how Python is being compiled/installed) than to setuptools itself (and therefore we can close this issue).

Successful example

rm -R isolated-test-in-container
mkdir isolated-test-in-container
cd isolated-test-in-container
new-item pyproject.toml
new-item Dockerfile

@"
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "mod"
version = "42"
"@ | add-content -Path pyproject.toml

@"
FROM python:3.11-windowsservercore-1809

RUN mkdir C:/pkg
WORKDIR C:/pkg
COPY pyproject.toml .

RUN python -m pip install -U pip setuptools wheel

CMD pip wheel -v .
"@ | add-content -Path Dockerfile
docker build -t isolated-test .
docker run --rm -it isolated-test

Output of `docker run`

```console Processing c:\pkg Running command pip subprocess to install build dependencies Collecting setuptools>=61 Using cached setuptools-65.5.1-py3-none-any.whl (1.2 MB) Installing collected packages: setuptools Successfully installed setuptools-65.5.1 Installing build dependencies ... done Running command Getting requirements to build wheel running egg_info creating mod.egg-info writing mod.egg-info\PKG-INFO writing dependency_links to mod.egg-info\dependency_links.txt writing top-level names to mod.egg-info\top_level.txt writing manifest file 'mod.egg-info\SOURCES.txt' reading manifest file 'mod.egg-info\SOURCES.txt' writing manifest file 'mod.egg-info\SOURCES.txt' Getting requirements to build wheel ... done Running command pip subprocess to install backend dependencies Collecting wheel Using cached wheel-0.38.4-py3-none-any.whl (36 kB) Installing collected packages: wheel Successfully installed wheel-0.38.4 Installing backend dependencies ... done Running command Preparing metadata (pyproject.toml) running dist_info creating C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info writing C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\PKG-INFO writing dependency_links to C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\dependency_link s.txt writing top-level names to C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\top_level.txt writing manifest file 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\SOURCES.txt' reading manifest file 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\SOURCES.txt' writing manifest file 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod.egg-info\SOURCES.txt' creating 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-modern-metadata-4yyajyko\mod-42.dist-info' Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: mod Running command Building wheel for mod (pyproject.toml) running bdist_wheel running build installing to build\bdist.win-amd64\wheel running install running install_egg_info running egg_info writing mod.egg-info\PKG-INFO writing dependency_links to mod.egg-info\dependency_links.txt writing top-level names to mod.egg-info\top_level.txt reading manifest file 'mod.egg-info\SOURCES.txt' writing manifest file 'mod.egg-info\SOURCES.txt' Copying mod.egg-info to build\bdist.win-amd64\wheel\.\mod-42-py3.11.egg-info running install_scripts creating build\bdist.win-amd64\wheel\mod-42.dist-info\WHEEL creating 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-wheel-uk07kl01\tmpm7j7vnk0\mod-42-py3-none-any.whl' and adding 'build\b dist.win-amd64\wheel' to it adding 'mod-42.dist-info/METADATA' adding 'mod-42.dist-info/WHEEL' adding 'mod-42.dist-info/top_level.txt' adding 'mod-42.dist-info/RECORD' removing build\bdist.win-amd64\wheel Building wheel for mod (pyproject.toml) ... done Created wheel for mod: filename=mod-42-py3-none-any.whl size=862 sha256=79a2c8146078715f5f6cf53f0a8fa88330938625d50f530e8711a4f6990da82a Stored in directory: C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-ephem-wheel-cache-dlh4ev4r\wheels\7a\c7\0a\dfe41c75aa52ca4e3 17402d3e007a60a5e660ca93de44e656d Successfully built mod ```

abravalheri commented 2 years ago

Please also note that I tried to run in a container the original suggestion (pip install nonnegcg - https://github.com/pypa/setuptools/issues/1631#issue-397021655) and that fails, but because of compilation problems, not because of underlying buffer has been detached. Instead, Python/pip/setuptools seem to be handling the stdout quite well:

rm -R isolated-test-in-container
mkdir isolated-test-in-container
cd isolated-test-in-container

new-item Dockerfile
@"
FROM python:3.11-windowsservercore-1809

RUN python -m pip install -U pip setuptools wheel numpy findblas Cython mkl-devel
CMD python -m pip install nonnegcg --no-build-isolation --no-use-pep517
"@ | add-content -Path Dockerfile
docker build -t isolated-test .
docker run --rm -it isolated-test

Output of `docker run`

```console Collecting nonnegcg Downloading nonnegcg-0.1.6.tar.gz (30 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: numpy in c:\python\lib\site-packages (from nonnegcg) (1.23.4) Collecting scipy Downloading scipy-1.9.3-cp311-cp311-win_amd64.whl (39.9 MB) ---------------------------------------- 39.9/39.9 MB 4.9 MB/s eta 0:00:00 Requirement already satisfied: cython in c:\python\lib\site-packages (from nonnegcg) (0.29.32) Requirement already satisfied: findblas in c:\python\lib\site-packages (from nonnegcg) (0.1.20) Building wheels for collected packages: nonnegcg Building wheel for nonnegcg (setup.py) ... error error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [46 lines of output] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-cpython-311 creating build\lib.win-amd64-cpython-311\nonnegcg copying nonnegcg\__init__.py -> build\lib.win-amd64-cpython-311\nonnegcg running egg_info writing nonnegcg.egg-info\PKG-INFO writing dependency_links to nonnegcg.egg-info\dependency_links.txt writing requirements to nonnegcg.egg-info\requires.txt writing top-level names to nonnegcg.egg-info\top_level.txt reading manifest file 'nonnegcg.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'nonnegcg.egg-info\SOURCES.txt' copying nonnegcg\nonnegcg.pxd -> build\lib.win-amd64-cpython-311\nonnegcg copying nonnegcg\pywrapper.pyx -> build\lib.win-amd64-cpython-311\nonnegcg running build_ext INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Optimized (vendor) Blas libraries are not found. Falls back to netlib Blas library which has worse performance. A better performance should be easily gained by switching Blas library. if self._calc_info(blas): C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. if self._calc_info(blas): C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. if self._calc_info(blas): WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. cythoning nonnegcg/pywrapper.pyx to nonnegcg\pywrapper.c C:\Python\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now ( Py2). This will change in a later release! File: C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-p4yynn08\nonnegcg_5d76d50f 1d9e4cef988095affeecd76b\nonnegcg\pywrapper.pyx tree = Parsing.p_module(s, pxd, full_module_name) building 'nonnegcg._minimize' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com /visual-cpp-build-tools/ [end of output] ERROR: Failed building wheel for nonnegcg Running setup.py clean for nonnegcg Failed to build nonnegcg Installing collected packages: scipy, nonnegcg Running setup.py install for nonnegcg ... error error: subprocess-exited-with-error × Running setup.py install for nonnegcg did not run successfully. │ exit code: 1 ╰─> [46 lines of output] running install C:\Python\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use buil d and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build\lib.win-amd64-cpython-311 creating build\lib.win-amd64-cpython-311\nonnegcg copying nonnegcg\__init__.py -> build\lib.win-amd64-cpython-311\nonnegcg running egg_info writing nonnegcg.egg-info\PKG-INFO writing dependency_links to nonnegcg.egg-info\dependency_links.txt writing requirements to nonnegcg.egg-info\requires.txt writing top-level names to nonnegcg.egg-info\top_level.txt reading manifest file 'nonnegcg.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'nonnegcg.egg-info\SOURCES.txt' copying nonnegcg\nonnegcg.pxd -> build\lib.win-amd64-cpython-311\nonnegcg copying nonnegcg\pywrapper.pyx -> build\lib.win-amd64-cpython-311\nonnegcg running build_ext INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Optimized (vendor) Blas libraries are not found. Falls back to netlib Blas library which has worse performance. A better performance should be easily gained by switching Blas library. if self._calc_info(blas): C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. if self._calc_info(blas): C:\Python\Lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. if self._calc_info(blas): WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. skipping 'nonnegcg\pywrapper.c' Cython extension (up-to-date) building 'nonnegcg._minimize' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com /visual-cpp-build-tools/ [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> nonnegcg note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. ```

dlaugt commented 2 years ago

Hi @abravalheri,

I have the same errors by executing the @anntzer's simple steps:

Windows 10 version 21H2
Install python 3.11.0 (which I have never installed before)
Create the pyproject.toml
Create the empty mod.py
From a cmd window, pip wheel -v .

The file mod-42-py3-none-any.whl is generated but there are these ugly error messages in the standard output like:

  --- Logging error ---
  Traceback (most recent call last):
    File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit
      stream.write(msg + self.terminator)
  ValueError: underlying buffer has been detached
  Call stack:
    File "C:\dev\Python311\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 351, in <module>
      main()
    File "C:\dev\Python311\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "C:\dev\Python311\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 249, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\build_meta.py", line 412, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\build_meta.py", line 397, in _build_with_temp_dir
      self.run_setup()
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 1, in <module>
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
      return run_commands(dist)
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
      dist.run_commands()
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands
      self.run_command(cmd)
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command
      super().run_command(command)
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command
      cmd_obj.run()
    File "C:\Users\daniel\AppData\Local\Temp\pip-build-env-pg7a2f7d\normal\Lib\site-packages\wheel\bdist_wheel.py", line 399, in run
      log.info(f"removing {self.bdist_dir}")
  Message: 'removing build\\bdist.win-amd64\\wheel'
  Arguments: ()

It is not linked to a python version as I have the same with python 3.9 and 3.10. I suspect something wrong in the logging of setuptools arround the following line? https://github.com/pypa/setuptools/blob/a0e8e53cecc238e3bce2247308fe7dd94114ef35/setuptools/dist.py#L1202-L1210

dlaugt commented 2 years ago

If it can help, here is the full traces: traces.log

dlaugt commented 2 years ago

Hi,

To be able to debug the use case, I have changed the steps as follow:

Windows 10 version 21H2
Install a fresh python 3.11.0
pip install setuptools
pip install wheel
Create empty mod.py
Create pyproject.toml
Create setup.py
python setup.py bdist_wheel

pyproject.toml:

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "mod"
version = "42"

setup.py:

from setuptools import setup
setup()

Output of `python setup.py bdist_wheel`

``` running bdist_wheel running build running build_py creating build creating build\lib copying mod.py -> build\lib running egg_info creating mod.egg-info writing mod.egg-info\PKG-INFO writing dependency_links to mod.egg-info\dependency_links.txt writing top-level names to mod.egg-info\top_level.txt writing manifest file 'mod.egg-info\SOURCES.txt' reading manifest file 'mod.egg-info\SOURCES.txt' writing manifest file 'mod.egg-info\SOURCES.txt' C:\dev\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 358, in run log.info(f"installing to {self.bdist_dir}") Message: 'installing to build\\bdist.win-amd64\\wheel' Arguments: () running install running install_lib creating build\bdist.win-amd64 creating build\bdist.win-amd64\wheel copying build\lib\mod.py -> build\bdist.win-amd64\wheel\. running install_egg_info Copying mod.egg-info to build\bdist.win-amd64\wheel\.\mod-42-py3.11.egg-info running install_scripts --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 379, in run self.write_wheelfile(distinfo_dir) File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 423, in write_wheelfile log.info(f"creating {wheelfile_path}") Message: 'creating build\\bdist.win-amd64\\wheel\\mod-42.dist-info\\WHEEL' Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 387, in run wf.write_files(archive_root) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 122, in write_files log.info(f"creating '{self.filename}' and adding '{base_dir}' to it") Message: "creating 'dist\\mod-42-py3-none-any.whl' and adding 'build\\bdist.win-amd64\\wheel' to it" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 387, in run wf.write_files(archive_root) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 137, in write_files self.write(path, arcname) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 153, in write self.writestr(zinfo, data, compress_type) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 165, in writestr log.info(f"adding '{fname}'") Message: "adding 'mod.py'" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 387, in run wf.write_files(archive_root) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 141, in write_files self.write(path, arcname) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 153, in write self.writestr(zinfo, data, compress_type) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 165, in writestr log.info(f"adding '{fname}'") Message: "adding 'mod-42.dist-info/METADATA'" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 387, in run wf.write_files(archive_root) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 141, in write_files self.write(path, arcname) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 153, in write self.writestr(zinfo, data, compress_type) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 165, in writestr log.info(f"adding '{fname}'") Message: "adding 'mod-42.dist-info/WHEEL'" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 387, in run wf.write_files(archive_root) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 141, in write_files self.write(path, arcname) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 153, in write self.writestr(zinfo, data, compress_type) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 165, in writestr log.info(f"adding '{fname}'") Message: "adding 'mod-42.dist-info/top_level.txt'" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 386, in run with WheelFile(wheel_path, "w", self.compression) as wf: File "C:\dev\Python311\Lib\zipfile.py", line 1342, in __exit__ self.close() File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 189, in close self.writestr(zinfo, data.getvalue()) File "C:\dev\Python311\Lib\site-packages\wheel\wheelfile.py", line 165, in writestr log.info(f"adding '{fname}'") Message: "adding 'mod-42.dist-info/RECORD'" Arguments: () --- Logging error --- Traceback (most recent call last): File "C:\dev\Python311\Lib\logging\__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: underlying buffer has been detached Call stack: File "C:\src\python\test\setup.py", line 3, in setup() File "C:\dev\Python311\Lib\site-packages\setuptools\__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands self.run_command(cmd) File "C:\dev\Python311\Lib\site-packages\setuptools\dist.py", line 1217, in run_command super().run_command(command) File "C:\dev\Python311\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command cmd_obj.run() File "C:\dev\Python311\Lib\site-packages\wheel\bdist_wheel.py", line 399, in run log.info(f"removing {self.bdist_dir}") Message: 'removing build\\bdist.win-amd64\\wheel' Arguments: () ```

The standard output is fixed if I change the following in setuptools/dist.py: from:

        sys.stdout = io.TextIOWrapper(
            sys.stdout.detach(), 'utf-8', errors, newline, line_buffering
        )
        try:
            return _Distribution.handle_display_options(self, option_order)
        finally:
            sys.stdout = io.TextIOWrapper(
                sys.stdout.detach(), encoding, errors, newline, line_buffering
            )

to:

        sys.stdout.reconfigure(encoding='utf-8')
        try:
            return _Distribution.handle_display_options(self, option_order)
        finally:
            sys.stdout.reconfigure(encoding=encoding)

Output of `python setup.py bdist_wheel`

``` running bdist_wheel running build running build_py creating build creating build\lib copying mod.py -> build\lib running egg_info creating mod.egg-info writing mod.egg-info\PKG-INFO writing dependency_links to mod.egg-info\dependency_links.txt writing top-level names to mod.egg-info\top_level.txt writing manifest file 'mod.egg-info\SOURCES.txt' reading manifest file 'mod.egg-info\SOURCES.txt' writing manifest file 'mod.egg-info\SOURCES.txt' C:\dev\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( installing to build\bdist.win-amd64\wheel running install running install_lib creating build\bdist.win-amd64 creating build\bdist.win-amd64\wheel copying build\lib\mod.py -> build\bdist.win-amd64\wheel\. running install_egg_info Copying mod.egg-info to build\bdist.win-amd64\wheel\.\mod-42-py3.11.egg-info running install_scripts creating build\bdist.win-amd64\wheel\mod-42.dist-info\WHEEL creating 'dist\mod-42-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it adding 'mod.py' adding 'mod-42.dist-info/METADATA' adding 'mod-42.dist-info/WHEEL' adding 'mod-42.dist-info/top_level.txt' adding 'mod-42.dist-info/RECORD' removing build\bdist.win-amd64\wheel ```

I will use this tweak. However, the function reconfigure() is available only from python 3.7. But that's fine for me as I'm using python 3.10.

dlaugt commented 2 years ago

From pypi website, the latest setuptools requires at least python 3.7. So, perhaps, this kind of tweak can be committed to the repo.

abravalheri commented 2 years ago

Hi @dlaugt, thank you very much for investigating this.

What is puzzling me is that I could not reproduce this in my Windows machine or in an isolated Windows container (I admit that I am probably not using the exact version of Windows you guys are using because of licenses/availability).

Probably there is some information missing to create a reproducer. I wonder which console/terminal program you guys are using and which encoding it is configured to use... Maybe that is the missing piece?

If you have tested the proposed patch and it works for you, would you like to submit a PR? Ideally we would like to have regression tests associated, but that might be tricky in this situation... Maybe just the fact that it does not fail the existing test suite is enough... (But please feel free to suggest a test suite if you are feeling creative). Once the PR is submitted, I will probably ask for the other maintainers to review.

dlaugt commented 2 years ago

I'm running this from a command prompt window (%windir%\system32\cmd.exe). The encoding is utf-8:

python -c "import sys; print(sys.stdout.encoding)"
utf-8

However, when I launch python setup.py bdist_wheel, it goes to setuptools.Distribution.handle_display_options(). Inside this function, the encoding is cp1252...

I will create a PR.

dlaugt commented 2 years ago

I guess I understood why you cannot reproduce it... In fact, I was redirecting the standard & error output into a file.

From cmd.exe, it worked fine when I run the following:

python -c "import sys; print(sys.stdout.encoding)"
utf-8

python setup.py bdist_wheel

From cmd.exe, it didn't work fine when I run the following:

python -c "import sys; print(sys.stdout.encoding)" > traces.log 2>&1
cp1252

python setup.py bdist_wheel > traces.log 2>&1

However, the command bdist_wheel should work also when you redirect the standard & error output into a file. So, the pull request is fixing the following use case: python setup.py bdist_wheel > traces.log 2>&1

pypa / setuptools

Windows installations crashing due to unavailability of standard output #1631

Successful example