search

yaml / pyyaml

Canonical source repository for PyYAML
MIT License
2.57k stars 517 forks source link

Installing the from source should not fail to find yaml.h on macos #662

Closed ssbarnea closed 2 years ago

ssbarnea commented 2 years ago

As libyaml is included with brew, the command below it should succeed in compiling it with libyaml and installing it:

$ pip3 install -v --no-binary :all: --user --force pyyaml
Using pip 22.2.2 from /Users/ssbarnea/.pyenv/versions/3.11-dev/lib/python3.11/site-packages/pip (python 3.11)
Collecting pyyaml
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Using cached setuptools-65.3.0.tar.gz (2.6 MB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting wheel
    Using cached wheel-0.37.1.tar.gz (66 kB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Collecting Cython
    Using cached Cython-0.29.32.tar.gz (2.1 MB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Skipping wheel build for wheel, due to binaries being disabled for it.
  Skipping wheel build for Cython, due to binaries being disabled for it.
  Building wheels for collected packages: setuptools
    Building wheel for setuptools (pyproject.toml): started
    Building wheel for setuptools (pyproject.toml): finished with status 'done'
    Created wheel for setuptools: filename=setuptools-65.3.0-py3-none-any.whl size=1232493 sha256=fea0e996d704d335e6929606fcb9a1e6ec76364059f6b609954f7a168364b6c0
    Stored in directory: /Users/ssbarnea/Library/Caches/pip/wheels/35/5a/9e/d0c34505588c5301e1dfe81ca3f80f5c1e9231655afc8447bb
  Successfully built setuptools
  Installing collected packages: wheel, setuptools, Cython
    Running setup.py install for wheel: started
    Running setup.py install for wheel: finished with status 'done'
    Running setup.py install for Cython: started
    Running setup.py install for Cython: finished with status 'done'
  Successfully installed Cython-0.29.32 setuptools-65.3.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing lib/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib/PyYAML.egg-info/top_level.txt
  cythoning yaml/_yaml.pyx to yaml/_yaml.c
  /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-build-env-nadj2cbv/overlay/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-install-pv0d4l6x/pyyaml_39c7df68c25949869bf1302178d0722d/yaml/_yaml.pxd
    tree = Parsing.p_module(s, pxd, full_module_name)
  reading manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'README'
  adding license file 'LICENSE'
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info
  writing /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/top_level.txt
  writing manifest file '/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/SOURCES.txt'
  skipping 'yaml/_yaml.c' Cython extension (up-to-date)
  reading manifest file '/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'README'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML.egg-info/SOURCES.txt'
  creating '/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-modern-metadata-6k3yk2p4/PyYAML-6.0.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pyyaml
  Running command Building wheel for pyyaml (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12.5-arm64-cpython-311
  creating build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/scanner.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/error.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/constructor.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/composer.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/events.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/__init__.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/representer.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/tokens.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/dumper.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/cyaml.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/parser.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/reader.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/loader.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/resolver.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/serializer.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/nodes.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  copying lib/yaml/emitter.py -> build/lib.macosx-12.5-arm64-cpython-311/yaml
  creating build/lib.macosx-12.5-arm64-cpython-311/_yaml
  copying lib/_yaml/__init__.py -> build/lib.macosx-12.5-arm64-cpython-311/_yaml
  running build_ext
  skipping 'yaml/_yaml.c' Cython extension (up-to-date)
  building 'yaml._yaml' extension
  creating build/temp.macosx-12.5-arm64-cpython-311
  creating build/temp.macosx-12.5-arm64-cpython-311/yaml
  ccache /usr/bin/cc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/ssbarnea/.pyenv/versions/3.11-dev/include/python3.11 -c yaml/_yaml.c -o build/temp.macosx-12.5-arm64-cpython-311/yaml/_yaml.o
  In file included from yaml/_yaml.c:744:
  yaml/_yaml.h:2:10: fatal error: 'yaml.h' file not found
  #include <yaml.h>
           ^~~~~~~~
  1 error generated.
  Error compiling module, falling back to pure Python
  installing to build/bdist.macosx-12.5-arm64/wheel
  running install
  running install_lib
  creating build/bdist.macosx-12.5-arm64
  creating build/bdist.macosx-12.5-arm64/wheel
  creating build/bdist.macosx-12.5-arm64/wheel/_yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/_yaml/__init__.py -> build/bdist.macosx-12.5-arm64/wheel/_yaml
  creating build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/scanner.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/error.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/constructor.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/composer.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/events.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/__init__.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/representer.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/tokens.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/dumper.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/cyaml.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/parser.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/reader.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/loader.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/resolver.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/serializer.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/nodes.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  copying build/lib.macosx-12.5-arm64-cpython-311/yaml/emitter.py -> build/bdist.macosx-12.5-arm64/wheel/yaml
  running install_egg_info
  running egg_info
  writing lib/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib/PyYAML.egg-info/top_level.txt
  reading manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'README'
  adding license file 'LICENSE'
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Copying lib/PyYAML.egg-info to build/bdist.macosx-12.5-arm64/wheel/PyYAML-6.0-py3.11.egg-info
  running install_scripts
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  creating build/bdist.macosx-12.5-arm64/wheel/PyYAML-6.0.dist-info/WHEEL
  creating '/private/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/pip-wheel-wq1zab0t/tmpdfq1mpfi/PyYAML-6.0-cp311-cp311-macosx_12_0_arm64.whl' and adding 'build/bdist.macosx-12.5-arm64/wheel' to it
  adding '_yaml/__init__.py'
  adding 'yaml/__init__.py'
  adding 'yaml/composer.py'
  adding 'yaml/constructor.py'
  adding 'yaml/cyaml.py'
  adding 'yaml/dumper.py'
  adding 'yaml/emitter.py'
  adding 'yaml/error.py'
  adding 'yaml/events.py'
  adding 'yaml/loader.py'
  adding 'yaml/nodes.py'
  adding 'yaml/parser.py'
  adding 'yaml/reader.py'
  adding 'yaml/representer.py'
  adding 'yaml/resolver.py'
  adding 'yaml/scanner.py'
  adding 'yaml/serializer.py'
  adding 'yaml/tokens.py'
  adding 'PyYAML-6.0.dist-info/LICENSE'
  adding 'PyYAML-6.0.dist-info/METADATA'
  adding 'PyYAML-6.0.dist-info/WHEEL'
  adding 'PyYAML-6.0.dist-info/top_level.txt'
  adding 'PyYAML-6.0.dist-info/RECORD'
  removing build/bdist.macosx-12.5-arm64/wheel
  Building wheel for pyyaml (pyproject.toml) ... done
  Created wheel for pyyaml: filename=PyYAML-6.0-cp311-cp311-macosx_12_0_arm64.whl size=45336 sha256=2d68abba0b541fcb6f203163ee9dbe440fb3c7ca56dabfc972d03e6ad4f1a306
  Stored in directory: /Users/ssbarnea/Library/Caches/pip/wheels/3c/de/35/2cbb99955fa449eeddb2b433b4369035ee5bdf0fc121035ffe
Successfully built pyyaml
Installing collected packages: pyyaml
  Attempting uninstall: pyyaml
    Found existing installation: PyYAML 6.0
    Uninstalling PyYAML-6.0:
      Removing file or directory /Users/ssbarnea/.cache/cpython/Users/ssbarnea/.local/lib/python3.11/site-packages/_yaml/
      Removing file or directory /Users/ssbarnea/.cache/cpython/Users/ssbarnea/.local/lib/python3.11/site-packages/yaml/
      Removing file or directory /Users/ssbarnea/.local/lib/python3.11/site-packages/PyYAML-6.0.dist-info/
      Removing file or directory /Users/ssbarnea/.local/lib/python3.11/site-packages/_yaml/
      Removing file or directory /Users/ssbarnea/.local/lib/python3.11/site-packages/yaml/
      Successfully uninstalled PyYAML-6.0
Successfully installed pyyaml-6.0

As we can see there was a failure to locate yaml.h, but I am looking locally and I can see it as being present in standard locations:

/opt/homebrew/include/yaml.h -> ../Cellar/libyaml/0.2.5/include/yaml.h
/opt/homebrew/lib/libyaml.a -> ../Cellar/libyaml/0.2.5/lib/libyaml.a

Solution

In order to prevent the build failure, user must define:

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

In order to avoid this manual step, we should automatically defined these if they are not already present.

nitzmahone commented 2 years ago

We've gotten burned enough trying to support Homebrew detection by default, only to break a bunch of other stuff in the process (or when homebrew changes something) that I'm kinda over that. The PyYAML setup code (most of which is over 20 years old!) has to get rewritten against modern setuptools pretty soon, since Cython 3.0 will irrecoverably break it. At that point, I'll also simplify and document the mechanisms to override build config for people that want to run the build themselves and bring their own libyaml (I don't really want to treat homebrew "specially" here, since it used to be Macports, and tomorrow it could be ?).

Meantime, the 3.11 wheels are up, so the original pressing need that triggered the ask should also be solved. :laughing:

ssbarnea commented 2 years ago

@nitzmahone A really big thanks for doing this! MacOS and pyyaml are both very important so we cannot really afford to ignore them. I will try to find some time after the summit to give hand with setuptools refactoring to modernize it, even if I do not have a lot of experience with cython stuff. I wonder if @webknjaz may be able to give us a hand here as I know he more experience compiling cython libs, maybe he already did this kind of upgrade for other library.

PS. Homebrew was a source of issues many years ago, but become much more stable these days. I think that last time I seen macports being used was like... a decade ago?