"AttributeError: cython_sources" with Cython 3.0.0a10

[The-Compiler]

When attempting to install PyYAML from sources with Cython 3.0.0a10, e.g. like so:

pip install --pre -v git+https://github.com/yaml/pyyaml.git

the install fails with:

Using pip 21.3.1 from /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting git+https://github.com/yaml/pyyaml.git
  Cloning https://github.com/yaml/pyyaml.git to /tmp/pip-req-build-rdv2sw86
  Running command git version
  git version 2.34.1
  Running command git clone --filter=blob:none -q https://github.com/yaml/pyyaml.git /tmp/pip-req-build-rdv2sw86
  Running command git rev-parse HEAD
  8cdff2c80573b8be8e8ad28929264a913a63aa33
  Resolved https://github.com/yaml/pyyaml.git to commit 8cdff2c80573b8be8e8ad28929264a913a63aa33
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /tmp/pip-standalone-pip-7sp3xayz/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-az2k859x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --pre -- setuptools wheel Cython
  Collecting setuptools
    Using cached setuptools-60.5.0-py3-none-any.whl (958 kB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting Cython
    Using cached Cython-3.0.0a10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.2 MB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-3.0.0a10 setuptools-60.5.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l
  running egg_info
  creating lib/PyYAML.egg-info
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 288, in <module>
      setup(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 312, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "setup.py", line 204, in get_source_files
      self.cython_sources(ext.sources, ext)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  Getting requirements to build wheel ... error
WARNING: Discarding git+https://github.com/yaml/pyyaml.git. Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.

I'm not sure if this is an intended change in Cython 3, or a bug - but given that "pyyaml setup is still relying on a bunch of ancient custom distutils bits" (#584) I thought I'd report it here first. Please let me know if you think I should rather report it there.

[nitzmahone]

It's definitely a regression between Cython 3.0.0a9 and a10; every previous 3.0.0 alpha has been able to build the pyyaml extension fine. This PR (included in a10) swapped out the base class on Cython's custom build_ext distutils command (which pyyaml further subclasses to customize some stuff). I haven't dug down all the way yet to see if it's something we should try to address in pyyaml's customization or if it's just a bug in the new base class.

Thanks for the report- regardless, we'll want to get this sorted one way or another, so I'll figure out where the problem is and work with the Cython/setuptools folks if necessary to get it working.

[tomplex]

I encountered this error when attempting to lock my dependencies with pipenv; I was able to fix it in my case by setting

[pipenv]
allow_prereleases = false

just figured I'd share this in case it helps someone else.

[nitzmahone]

allow_prereleases = false

That'll only work until Cython 3.0 releases, unless we're able to get a change made over there (which it sounds like they're amenable to, if it's not too gnarly)

[RobertoPrevato]

I don't know if this can be of interest for anyone reading here, but I had the same issue in a GitHub Workflow, only happening on Python 3.10. The failed build can be seen here: https://github.com/Neoteroi/BlackSheep/runs/5098546591?check_suite_focus=true

In my case, PyYAML is a dependency and I was testing another wheel with this command:

pip install --pre blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

In my case, removing the --pre option fixed the problem.

pip install blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

[nitzmahone]

Yep, unfortunately --pre seems to affect all deps and build deps as well under pip.

[mgorny]

Ping. Cython 3.0.0 has already reached beta releases, so we need some solution sooner than later.

[nitzmahone]

Cool- we've been poking at doing another release, and since we sat on it long enough, setuptools and pip now fully support PEP517/518 config passthru from the CLI, which should make the replacement of the dynamic bits a whole lot easier.

[jmakov]

Stumbled upon this bug while trying to install Pycaret:

Collecting PyYAML~=5.1                                                                                                                   
#0 30.24   Downloading PyYAML-5.4.1.tar.gz (175 kB)                                                                                               #0 30.24      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 34.3 MB/s eta 0:00:00
#0 30.32   Installing build dependencies: started                                                                                                 
#0 33.08   Installing build dependencies: finished with status 'done'                                                                             
#0 33.08   Getting requirements to build wheel: started                  
#0 33.19   Getting requirements to build wheel: finished with status 'error'                                                                      #0 33.20   error: subprocess-exited-with-error                                                                                                    
#0 33.20                                                                                                                                          
#0 33.20   × Getting requirements to build wheel did not run successfully.
#0 33.20   │ exit code: 1                                                                                                                         
#0 33.20   ╰─> [50 lines of output]                                      
#0 33.20       /tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:516: SetuptoolsDeprecationWarning: T
he license_file parameter is deprecated, use license_files instead.
#0 33.20         warnings.warn(msg, warning_class)                                                                                                
#0 33.20       running egg_info                                          
#0 33.20       writing lib3/PyYAML.egg-info/PKG-INFO                                                                                              
#0 33.20       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt                                                              
#0 33.20       writing top-level names to lib3/PyYAML.egg-info/top_level.txt                                                                      
#0 33.20       Traceback (most recent call last):
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <
module>
#0 33.20           main()
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in m
ain
#0 33.20           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in g
et_requires_for_build_wheel
#0 33.20           return hook(config_settings)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_b
uild_wheel
#0 33.20           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_require
s
#0 33.20           self.run_setup()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
#0 33.20           exec(code, locals())
#0 33.20         File "<string>", line 271, in <module>
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
#0 33.20           return distutils.core.setup(**attrs)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 33.20           return run_commands(dist)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 33.20           dist.run_commands()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 33.20           self.run_command(cmd)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
#0 33.20           super().run_command(command)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 33.20           cmd_obj.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
#0 33.20           self.find_sources()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
#0 33.20           mm.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 570, in run
#0 33.20           self.add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 608, in add_defaults
#0 33.20           sdist.add_defaults(self)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
#0 33.20           super().add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 33.20           self._add_defaults_ext()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 33.20           self.filelist.extend(build_ext.get_source_files())
#0 33.20         File "<string>", line 201, in get_source_files
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 33.20           raise AttributeError(attr)
#0 33.20       AttributeError: cython_sources
#0 33.20       [end of output]

[nitzmahone]

@jmakov the likely short-term fix will be to pin the Cython PEP518 build deps to pre-3.0 Cython in a 6.x release (see #702), but we generally consider the 5.x release branch dead, so hadn't planned on re-publishing the build change back there. You might want to try and get the Pycaret folks (or whoever's ultimately pinning PyYAML to ~=5.1) to update their deps.

[jmakov]

@nitzmahone thank you so much for pointing out the solution! Will link your answer to an issue in their repo.

[nitzmahone]

@jmakov after looking at the issue you just linked, now I think I see how you ended up building PyYAML with an unreleased Cython 3.x; IIRC pip's --pre option globally allows pre-releases to be considered for installation (so for the package you asked for, as well as its entire dep tree and any build requirements pip installs).

If you just ask pip for for a specific pre-release version of Pycaret (without using the --pre flag, eg pip install pycaret==3.0.0rc8), the normal runtime and build-dep rules will be applied and PyYAML shouldn't try to build with an unreleased Cython.

[jmakov]

@nitzmahone that's a very interesting gotcha! This also explains why this works, thanks!

[mgorny]

This is now even more urgent as Cython<3 doesn't support Python 3.12.

[nitzmahone]

@mgorny where are you hearing that? Empirically speaking, Cython 0.29.x has been working flawlessly on 3.12- PyYAML extension builds have been passing the full test suite all along through the current Python 3.12.0b1.

(not that I don't want a better excuse to spend the time shredding PyYAML's creaky old packaging code to work under Cython3 while still doing everything it needs to do :laughing: )

[thesamesam]

@nitzmahone That's my bad, I think! The tests failed immediately (try to import imp, keeping an eye on https://github.com/cython/cython/issues/5285) but I thought it'd failed to compile too.

I'm going to test a bunch of reverse dependencies like pyyaml in Gentoo and unleash it if it works. Apologies for the error!

EDIT: Done!

[pinghedm]

Any update on this, now that Cython 3.0 has been released?

[Kaelten]

As with all the mentions, this is breaking our ability to build our production apps.

Is there any chance of getting a release that pins Cython? Failing that does anyone know how to work around this by pinning Cython when using poetry?

Edit: I've realized that this is happening for PyYaml 6.0 too

[pinghedm]

Our workaround for the time being is, in our build process, to install a compatible cython first and then use --no-build-isolation. Eg

pip install "Cython<3.0" pyyaml --no-build-isolation # new line
pip install -r requirements.txt # original build step, which has pyyaml in it

if that helps anyone

[MisterGlass]

This started effecting our builds this morning. In our case we had pinned pyyaml to 5.x, so I just had to upgrade to 6.x and things seem to be working.

[kvaggelakos]

Just building on @pinghedm's solution. If you happen to have another package that depends on pyyaml 5.x (like awscli). You can install it like this:

pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
pip install -r requirements.txt

[rosskarchner]

@MisterGlass Are you using Alpine, by any chance?

[muleyashutosh]

^^above not working for me

Here's my dockerFile

FROM python:3.10-alpine

RUN pip install "Cython<3.0" pyyaml --no-build-isolation
RUN pip install -r requirements.txt

• requirements.txt

  PyGithub==1.57
  PyYAML==6.0

[matt-codecov]

we're already on pyyaml 6.0 but are affected by this issue. we're on python:3.9.16-alpine3.17 and @pinghedm's workaround unblocks us.

[muleyashutosh]

this is the error that I get after trying @pinghedm 's workaround

.80 Collecting pycparser
18.82   Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
19.77      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 122.1 kB/s eta 0:00:00
19.79 Building wheels for collected packages: cffi
19.79   Building wheel for cffi (setup.py): started
20.08   Building wheel for cffi (setup.py): finished with status 'error'
20.09   error: subprocess-exited-with-error
20.09   
20.09   × python setup.py bdist_wheel did not run successfully.
20.09   │ exit code: 1
20.09   ╰─> [47 lines of output]
20.09       
20.09           No working compiler found, or bogus compiler options passed to
20.09           the compiler from Python's standard "distutils" module.  See
20.09           the error messages above.  Likely, the problem is not related
20.09           to CFFI but generic to the setup.py of any Python package that
20.09           tries to compile C code.  (Hints: on OS/X 10.8, for errors about
20.09           -mno-fused-madd see http://stackoverflow.com/questions/22313407/
20.09           Otherwise, see https://wiki.python.org/moin/CompLangPython or
20.09           the IRC channel #python on irc.libera.chat.)
20.09       
20.09           Trying to continue anyway.  If you are trying to install CFFI from
20.09           a build done in a different context, you can ignore this warning.
20.09       
20.09       /usr/local/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
20.09         warnings.warn(msg, warning_class)
20.09       running bdist_wheel
20.09       running build
20.09       running build_py
20.09       creating build
20.09       creating build/lib.linux-aarch64-cpython-310
20.09       creating build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/__init__.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/error.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/lock.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/api.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/model.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-310/cffi
20.09       copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-310/cffi
20.09       running build_ext
20.09       building '_cffi_backend' extension
20.09       creating build/temp.linux-aarch64-cpython-310
20.09       creating build/temp.linux-aarch64-cpython-310/c
20.09       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.10 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-310/c/_cffi_backend.o
20.09       error: command 'gcc' failed: No such file or directory
20.09       [end of output]
20.09   
20.09   note: This error originates from a subprocess, and is likely not a problem with pip.
20.09   ERROR: Failed building wheel for cffi
20.09   Running setup.py clean for cffi
20.21 Failed to build cffi
20.25 Installing collected packages: wrapt, urllib3, pyjwt, pycparser, idna, charset-normalizer, certifi, requests, deprecated, cffi, pynacl, PyGithub
20.54   Running setup.py install for cffi: started
20.69   Running setup.py install for cffi: finished with status 'error'
20.70   error: subprocess-exited-with-error
20.70   
20.70   × Running setup.py install for cffi did not run successfully.
20.70   │ exit code: 1
20.70   ╰─> [49 lines of output]
20.70       
20.70           No working compiler found, or bogus compiler options passed to
20.70           the compiler from Python's standard "distutils" module.  See
20.70           the error messages above.  Likely, the problem is not related
20.70           to CFFI but generic to the setup.py of any Python package that
20.70           tries to compile C code.  (Hints: on OS/X 10.8, for errors about
20.70           -mno-fused-madd see http://stackoverflow.com/questions/22313407/
20.70           Otherwise, see https://wiki.python.org/moin/CompLangPython or
20.70           the IRC channel #python on irc.libera.chat.)
20.70       
20.70           Trying to continue anyway.  If you are trying to install CFFI from
20.70           a build done in a different context, you can ignore this warning.
20.70       
20.70       /usr/local/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
20.70         warnings.warn(msg, warning_class)
20.70       running install
20.70       /usr/local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
20.70         warnings.warn(
20.70       running build
20.70       running build_py
20.70       creating build
20.70       creating build/lib.linux-aarch64-cpython-310
20.70       creating build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/__init__.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/error.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/lock.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/api.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/model.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-310/cffi
20.70       copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-310/cffi
20.70       running build_ext
20.70       building '_cffi_backend' extension
20.70       creating build/temp.linux-aarch64-cpython-310
20.70       creating build/temp.linux-aarch64-cpython-310/c
20.70       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.10 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-310/c/_cffi_backend.o
20.70       error: command 'gcc' failed: No such file or directory
20.70       [end of output]
20.70   
20.70   note: This error originates from a subprocess, and is likely not a problem with pip.
20.70 error: legacy-install-failure
20.70 
20.70 × Encountered error while trying to install package.
20.70 ╰─> cffi
20.70 
20.70 note: This is an issue with the package mentioned above, not pip.
20.70 hint: See above for output from the failure.
20.70 
20.70 [notice] A new release of pip is available: 23.0.1 -> 23.2
20.70 [notice] To update, run: pip install --upgrade pip
------
Dockerfile:7
--------------------
   5 |     # Install dependencies
   6 |     RUN pip install "Cython<3.0" pyyaml --no-build-isolation
   7 | >>> RUN pip install -r requirements.txt
   8 |     
   9 |     # Set the entrypoint
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1

[rosskarchner]

Same as @muleyashutosh -- the complete Dockerfile is:

FROM python:alpine

RUN apk add bash
RUN pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
RUN pip wheel install --no-cache-dir cfn-lint==0.78.1

[Dominick-Peluso-Bose]

@muleyashutosh

error: command 'gcc' failed: No such file or directory

Install gcc and other required build dependencies before doing the pip install (apk add gcc etc)

[jimbojd72]

Workaound doesn't work for me as well (no gcc error though)

FROM python:3.11.4-alpine3.18

RUN apk add \
    bash \
    conntrack-tools \
    iptables \
 && rm -rf /var/cache/apk/*

WORKDIR /usr/src/kamailio-metrics/

COPY requirements.txt .
COPY pystn ./pystn

RUN python3 -m venv .venv && \
    source .venv/bin/activate && \
    pip install --upgrade pip && \
    pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
    pip wheel install --no-cache-dir cfn-lint==0.78.1

❯ make build &&  make push
docker build -t jcr.io/platform/kamailio-metrics:jdesmarais .
[+] Building 10.0s (10/11)                                                                                                                                                                                             docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.06kB                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3.11.4-alpine3.18                                                                                                                                                            0.8s
 => [1/7] FROM docker.io/library/python:3.11.4-alpine3.18@sha256:25df32b602118dab046b58f0fe920e3301da0727b5b07430c8bcd4b139627fdc                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                      0.0s
 => => transferring context: 3.78kB                                                                                                                                                                                                    0.0s
 => CACHED [2/7] RUN apk add     bash     conntrack-tools     iptables  && rm -rf /var/cache/apk/*                                                                                                                                     0.0s
 => CACHED [3/7] WORKDIR /usr/src/kamailio-metrics/                                                                                                                                                                                    0.0s
 => CACHED [4/7] COPY requirements.txt .                                                                                                                                                                                               0.0s
 => CACHED [5/7] COPY pystn ./pystn                                                                                                                                                                                                    0.0s
 => ERROR [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1   9.1s
------
 > [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt:
4.962 Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.1.2)
5.115 Collecting pip
5.379   Downloading pip-23.2-py3-none-any.whl (2.1 MB)
5.611      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 9.1 MB/s eta 0:00:00
5.676 Installing collected packages: pip
5.676   Attempting uninstall: pip
5.679     Found existing installation: pip 23.1.2
5.727     Uninstalling pip-23.1.2:
5.734       Successfully uninstalled pip-23.1.2
7.056 Successfully installed pip-23.2
8.028 Collecting Cython<3.0
8.028   Obtaining dependency information for Cython<3.0 from https://files.pythonhosted.org/packages/cf/a3/43bf5b3ba528a8fa2aa25dd48317e493b46ce838098605277b9efc7a0c1a/Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata
8.264   Downloading Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (3.1 kB)
8.333 Collecting pyyaml<6
8.377   Downloading PyYAML-5.4.1.tar.gz (175 kB)
8.479      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 1.8 MB/s eta 0:00:00
8.651   Preparing metadata (pyproject.toml): started
8.929   Preparing metadata (pyproject.toml): finished with status 'error'
8.935   error: subprocess-exited-with-error
8.935
8.935   × Preparing metadata (pyproject.toml) did not run successfully.
8.935   │ exit code: 1
8.935   ╰─> [14 lines of output]
8.935       /usr/src/kamailio-metrics/.venv/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
8.935         warnings.warn(msg, warning_class)
8.935       running dist_info
8.935       creating /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info
8.935       writing /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/PKG-INFO
8.935       writing dependency_links to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/dependency_links.txt
8.935       writing top-level names to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/top_level.txt
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest template 'MANIFEST.in'
8.935       adding license file 'LICENSE'
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       creating '/tmp/pip-modern-metadata-i70halss/PyYAML-5.4.1.dist-info'
8.935       error: invalid command 'bdist_wheel'
8.935       [end of output]
8.935
8.935   note: This error originates from a subprocess, and is likely not a problem with pip.
8.938 error: metadata-generation-failed
8.938
8.938 × Encountered error while generating package metadata.
8.938 ╰─> See above for output.
8.938
8.938 note: This is an issue with the package mentioned above, not pip.
8.938 hint: See above for details.
------
Dockerfile:38
--------------------
  37 |
  38 | >>> RUN python3 -m venv .venv && \
  39 | >>>     source .venv/bin/activate && \
  40 | >>>     pip install --upgrade pip && \
  41 | >>>     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
  42 | >>>     pip wheel install --no-cache-dir cfn-lint==0.78.1 && \
  43 | >>>     pip install -r requirements.txt
  44 |
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install \"Cython<3.0\" \"pyyaml<6\" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt" did not complete successfully: exit code: 1

[MisterGlass]

@MisterGlass Are you using Alpine, by any chance?

No, I'm using python:3.11 as a base

[tjs-intel]

@jimbojd72 I had the same problem and the resolution was to pip install wheel before the pip install "Cython<3.0" ... command

[romilbhardwaj]

For folks relying on aws-cli, pinning pyyaml<=5.3.1 helped us.

NOTE - please read this note from aws-cli maintainers before you use pyyaml<=5.3.1:

We do not recommend installing an older version of PyYAML as PyYAML version 5.3.1 is associated with CVE-2020-14343 that was fixed in version 5.4.

[stefansedich]

For folks relying on aws-cli, pinning pyyaml<=5.3.1 helped us.

Thanks @romilbhardwaj that did the trick for us too!

[rosskarchner]

adding pip install wheel fixed it for us

FROM python:alpine

RUN apk add bash
RUN pip install wheel
RUN pip install "Cython<3.0" "pyyaml<6" --no-build-isolation
RUN pip install --no-cache-dir cfn-lint==0.65.1

[muleyashutosh]

this fixed it for me.

RUN apk update && apk add build-base libffi-dev
RUN pip install "Cython<3.0" pyyaml --no-build-isolation
RUN pip install -r requirements.txt

[r-asiebert]

Reporting identical issues in an Ubuntu 20.04 container running Python 3.10.1 compiled from source + pip 21.3.

cython compilation error specifically with pyyaml==5.4.1, installs fine with latest (6.0) or 5.3.1.

Sad morning to discover PEP 518 (--no-build-isolation). So much for freezing my requirements.txt 😕

[nateprewitt]

@nitzmahone is there anyway we can get https://github.com/yaml/pyyaml/pull/702 merged and released in 6.x to help stem the current build issues? There unfortunately isn't a robust way for downstream packages to mitigate this without defining a runtime dependency on Cython for constraint.

[willcharlton]

Is there any way to specify --no-build-isolation inside of a setup.cfg or requirements.txt file?

[jimbojd72]

Workaound doesn't work for me as well (no gcc error though)

FROM python:3.11.4-alpine3.18

RUN apk add \
    bash \
    conntrack-tools \
    iptables \
 && rm -rf /var/cache/apk/*

WORKDIR /usr/src/kamailio-metrics/

COPY requirements.txt .
COPY pystn ./pystn

RUN python3 -m venv .venv && \
    source .venv/bin/activate && \
    pip install --upgrade pip && \
    pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
    pip wheel install --no-cache-dir cfn-lint==0.78.1

❯ make build &&  make push
docker build -t jcr.io/platform/kamailio-metrics:jdesmarais .
[+] Building 10.0s (10/11)                                                                                                                                                                                             docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.06kB                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3.11.4-alpine3.18                                                                                                                                                            0.8s
 => [1/7] FROM docker.io/library/python:3.11.4-alpine3.18@sha256:25df32b602118dab046b58f0fe920e3301da0727b5b07430c8bcd4b139627fdc                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                      0.0s
 => => transferring context: 3.78kB                                                                                                                                                                                                    0.0s
 => CACHED [2/7] RUN apk add     bash     conntrack-tools     iptables  && rm -rf /var/cache/apk/*                                                                                                                                     0.0s
 => CACHED [3/7] WORKDIR /usr/src/kamailio-metrics/                                                                                                                                                                                    0.0s
 => CACHED [4/7] COPY requirements.txt .                                                                                                                                                                                               0.0s
 => CACHED [5/7] COPY pystn ./pystn                                                                                                                                                                                                    0.0s
 => ERROR [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1   9.1s
------
 > [6/7] RUN python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt:
4.962 Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.1.2)
5.115 Collecting pip
5.379   Downloading pip-23.2-py3-none-any.whl (2.1 MB)
5.611      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 9.1 MB/s eta 0:00:00
5.676 Installing collected packages: pip
5.676   Attempting uninstall: pip
5.679     Found existing installation: pip 23.1.2
5.727     Uninstalling pip-23.1.2:
5.734       Successfully uninstalled pip-23.1.2
7.056 Successfully installed pip-23.2
8.028 Collecting Cython<3.0
8.028   Obtaining dependency information for Cython<3.0 from https://files.pythonhosted.org/packages/cf/a3/43bf5b3ba528a8fa2aa25dd48317e493b46ce838098605277b9efc7a0c1a/Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata
8.264   Downloading Cython-0.29.36-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (3.1 kB)
8.333 Collecting pyyaml<6
8.377   Downloading PyYAML-5.4.1.tar.gz (175 kB)
8.479      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 1.8 MB/s eta 0:00:00
8.651   Preparing metadata (pyproject.toml): started
8.929   Preparing metadata (pyproject.toml): finished with status 'error'
8.935   error: subprocess-exited-with-error
8.935
8.935   × Preparing metadata (pyproject.toml) did not run successfully.
8.935   │ exit code: 1
8.935   ╰─> [14 lines of output]
8.935       /usr/src/kamailio-metrics/.venv/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
8.935         warnings.warn(msg, warning_class)
8.935       running dist_info
8.935       creating /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info
8.935       writing /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/PKG-INFO
8.935       writing dependency_links to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/dependency_links.txt
8.935       writing top-level names to /tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/top_level.txt
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       reading manifest template 'MANIFEST.in'
8.935       adding license file 'LICENSE'
8.935       writing manifest file '/tmp/pip-modern-metadata-i70halss/PyYAML.egg-info/SOURCES.txt'
8.935       creating '/tmp/pip-modern-metadata-i70halss/PyYAML-5.4.1.dist-info'
8.935       error: invalid command 'bdist_wheel'
8.935       [end of output]
8.935
8.935   note: This error originates from a subprocess, and is likely not a problem with pip.
8.938 error: metadata-generation-failed
8.938
8.938 × Encountered error while generating package metadata.
8.938 ╰─> See above for output.
8.938
8.938 note: This is an issue with the package mentioned above, not pip.
8.938 hint: See above for details.
------
Dockerfile:38
--------------------
  37 |
  38 | >>> RUN python3 -m venv .venv && \
  39 | >>>     source .venv/bin/activate && \
  40 | >>>     pip install --upgrade pip && \
  41 | >>>     pip install "Cython<3.0" "pyyaml<6" --no-build-isolation && \
  42 | >>>     pip wheel install --no-cache-dir cfn-lint==0.78.1 && \
  43 | >>>     pip install -r requirements.txt
  44 |
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m venv .venv &&     source .venv/bin/activate &&     pip install --upgrade pip &&     pip install \"Cython<3.0\" \"pyyaml<6\" --no-build-isolation &&     pip wheel install --no-cache-dir cfn-lint==0.78.1 &&     pip install -r requirements.txt" did not complete successfully: exit code: 1

Building with pyyaml pinned to version 5.3.1 fixed this issue for me

RUN python3 -m venv .venv && \
    source .venv/bin/activate && \
    pip install --upgrade pip && \
    pip install 'pyyaml<=5.3.1' && \
    pip install wheel && \
    pip install -r requirements.txt

[r-asiebert]

@nateprewitt

nitzmahone is there anyway we can get #702 merged and released in 6.x to help stem the current build issues? There unfortunately isn't a robust way for downstream packages to mitigate this without defining a runtime dependency on Cython for constraint.

Are you running into this issue with 6.0? 5.4.1 blows up for me, but a fresh virtualenv has no problem with pip install pyyaml==6.0

[kylebarron]

There unfortunately isn't a robust way for downstream packages to mitigate this without defining a runtime dependency on Cython for constraint.

For some package managers, e.g. Poetry, even defining a Cython runtime dependency doesn't help because Poetry builds wheels for sdist packages upon installation in a temporary directory that's isolated from the runtime environment and only includes packages defined in PEP 517's build-requires

[nateprewitt]

@r-asiebert

The specific issue here is if you're trying to install PyYAML on a system that doesn't have a wheel available. Cython is only involved when building from source. Most of the report are on Alpine because there aren't musl wheels available. If 6.x is working, my guess is you're using Python 3.10 or 3.11 which didn't have wheels available in the 5.4.1 release.

This will continue to be an issue until the build dependencies are properly constrained.

[MauDantas]

Anyone using poetry? It seems I only can use 6 version with it.

Does anyone had the same problem?

[MauDantas]

@r-asiebert

The specific issue here is if you're trying to install PyYAML on a system that doesn't have a wheel available. Cython is only involved when building from source. Most of the report are on Alpine because there aren't musl wheels available. If 6.x is working, my guess is you're using Python 3.10 or 3.11 which didn't have wheels available in the 5.4.1 release.

This will continue to be an issue until the build dependencies are properly constrained.

I'm using python 11 on alpine on a Fresh environment using Poetry... But, as mentioned by kylebarron, this affects me anyways...

[nateprewitt]

@MauDantas

Most of the report are on Alpine because there aren't musl wheels available.

All Alpine images are affected.

[henryiii]

This is breaking all my testing for 3.12, which doesn't have wheels (they can be uploaded pretty safely now with 3.12.0b4).

You can set PIP_CONSTRAINTS_FILE to a file with a constraint on pyyaml that will lock an older version if that helps. Or on Cython, which would help, I think. I'm not sure Poetry will respect it, but most tools use pip which will.

[Kaelten]

For my poetry based projects building in docker I was able to work around this by doing the following in my docker file:

RUN  poetry run pip install "Cython<3.0" pyyaml --no-build-isolation && \
     poetry install

[phillipuniverse]

@Kaelten thanks for the suggestion! That works for me on Poetry 1.5.1!

[b-enoit-be]

@r-asiebert Are you running into this issue with 6.0?

It does, at least for me (I cut in the stack trace for brevity):

$ docker run --rm python:alpine python -m pip install PyYAML 
Collecting PyYAML
  Downloading PyYAML-6.0.tar.gz (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.0/125.0 kB 4.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        (...)
        File "/tmp/pip-build-env-n1npha3u/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[mlissner]

In poetry, we were able to fix this by updating pyyaml to 6.0.

poetry add pyyaml@latest

That seemed to do the trick.

[mharding-hpe]

I would dearly love to see https://github.com/yaml/pyyaml/pull/702 get merged and released as 6.0.1, as a workaround to this issue, at least for folks using version 6.0.0

[alexandruvesa]

Does anybody solved the issue for python:3.9-slim-bullseye docker image?