search

yaml / pyyaml

Canonical source repository for PyYAML
MIT License
2.47k stars 507 forks source link

Error installing Pyyaml==5.4, Cython_sources #724

Closed MeliJuanmi closed 11 months ago

MeliJuanmi commented 11 months ago

I am tyring to install the 5.4 version, but I got the following output:

`Collecting pyyaml==5.4 Using cached PyYAML-5.4.tar.gz (174 kB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [68 lines of output] /private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg !!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.

          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************

  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/Users/uangutierrez/.fury/fury_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/uangutierrez/.fury/fury_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/uangutierrez/.fury/fury_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<string>", line 201, in get_source_files
    File "/private/var/folders/jq/gc3kdhbj0tg3r798nj8wlgl86xxhf9/T/pip-build-env-qbudtvrl/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.`

94rain commented 11 months ago

I just had the same issue with pyyaml 6.0.0

Duplicate of https://github.com/yaml/pyyaml/issues/723?

mcdonnnj commented 11 months ago

Both are duplicates of https://github.com/yaml/pyyaml/issues/601. This has been on the horizon for a long time apparently.

MeliJuanmi commented 11 months ago

Yes, but I see it is again backup discussion, just 50 minutes ago !

wesleykendall commented 11 months ago

Yea seems like now things are going to break more loudly for others since Cython3 was released.

AlexDld commented 11 months ago

You can use PyYaml 5.3.1 until the issue is resolved.

henryiii commented 11 months ago

This has broken Python 3.12 as well; there aren't pre-built wheels for 3.12 yet (ABI is now supposed to be stable as of beta 4, so you can add them ;) )

Setting:

"pyyaml!=6.0.0,!=5.4.0,!=5.4.1", # pyyaml is broken with cython 3

Does work for now on 3.12.

AndrewCharlesHay commented 11 months ago

You can use PyYaml 5.3.1 until the issue is resolved.

@AlexDld Thank you! Worked for me!

olliemath commented 11 months ago

Affecting us too and our security policy won't let us downgrade to 5.3 because of pre-5.4 vulnerabilities

But pip install "cython<3.0.0" && pip install --no-build-isolation pyyaml==6.0 did work (as per the linked issue)

tschaffter commented 11 months ago

We are experiencing the same issue today with pyyaml@5.4.1. What I don't understand yet is why we were able to install this version on Friday and not today? What has changed since Friday?

On Friday:

image

Today: Failed to install /home/vscode/.cache/pypoetry/artifacts/b6/23/45/f5dfdd6e8ba0f620504858ddeb20b47f50b03d0c4b18f873f6575d2e78/PyYAML-5.4.1.tar.gz

henryiii commented 11 months ago

Cython 3.0 came out since Friday.

johnthagen commented 11 months ago

Cython 3 was released 4 hours ago: https://pypi.org/project/Cython/3.0.0/#history

This coincides with when our PyYAML 6.0.0 installs via Poetry in Alpine Linux containers started failing. 😢

andy-maier commented 11 months ago

We cannot use PyYAML 5.3 due to dependencies requiring 5.4. On Python 3.10+3.11, using PyYAML 6.0 also works, because it provides wheel archives for these Python versions.

Is there a way to have PyYAML use Cython<3 for its installation?

andy-maier commented 11 months ago

Based on the available wheel archives of PyYAML, the following requirements work. These are designed such that the highest possible working version is used that has a wheel archive, so that Cython is not used during installation of PyYAML:

PyYAML>=5.3.1; python_version <= '3.5'
PyYAML>=5.3.1,!=5.4.0,!=5.4.1; python_version >= '3.6' and python_version <= '3.11'
PyYAML>=5.3.1,!=5.4.0,!=5.4.1,!=6.0.0; python_version >= '3.12'
sbtaylor15 commented 11 months ago

Here is the work around I am using for PyYaml in an Alpine Docker image.

  1. apk add --no-cache py3-yaml
  2. set PYTHONPATH to point to the 'pre-built' binary for PyYaml. PYTHONPATH=/usr/lib/python3.11/site-packages
  3. pip install pyyaml will return Requirement already satisfied and skip the build from source that fails.

Note: Anything that depends on the pyyaml will find the 'pre-built' binary and use that instead of building from source.

DockerFile

FROM python:3.11-alpine

# Set the search location to include pre-built binary modules
ENV PYTHONPATH /usr/lib/python3.11/site-packages

# Install the binary version of PyYaml
RUN apk add --no-cache py3-yaml

# Installing via pip will return dependency aleady satistifed
RUN pip install pyyaml

Docker Build Output

#1 [internal] load build definition from Dockerfile
#1 sha256:cc3f81718a377174575824bcb3eb33e8d90eeaa1eb08b2d9713beb668a9ce703
#1 transferring dockerfile: 47B 0.0s done
#1 DONE 0.2s

#2 [internal] load .dockerignore
#2 sha256:d32b232ac1f861e46756b7e44143e2ecee45a2372212b9a3e81dc9ddea7b40a3
#2 transferring context: 2B 0.0s done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/python:3.11-alpine
#3 sha256:8dcc1f1a926b4737e2595112cab17d76e100c8bd934bb54cd42e3c56611a8544
#3 DONE 0.8s

#4 [1/3] FROM docker.io/library/python:3.11-alpine@sha256:25df32b602118dab046b58f0fe920e3301da0727b5b07430c8bcd4b139627fdc
#4 sha256:ae77d191f15eaea9468e679d1fef42b898972a05a8974324454f8c978a42ee58
#4 CACHED

#5 [2/3] RUN apk add --no-cache py3-yaml
#5 sha256:5fa9c62004da1632ec1b7a1855f41ecac63fa1df5b5536c81a412c4bb8ab6a45
#5 0.883 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
#5 1.980 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
#5 5.189 (1/10) Installing libgcc (12.2.1_git20220924-r10)
#5 5.297 (2/10) Installing libstdc++ (12.2.1_git20220924-r10)
#5 6.380 (3/10) Installing mpdecimal (2.5.1-r2)
#5 6.557 (4/10) Installing python3 (3.11.4-r0)
#5 19.65 (5/10) Installing python3-pycache-pyc0 (3.11.4-r0)
#5 25.65 (6/10) Installing pyc (0.1-r0)
#5 25.69 (7/10) Installing py3-yaml-pyc (6.0-r3)
#5 25.87 (8/10) Installing python3-pyc (3.11.4-r0)
#5 25.91 (9/10) Installing yaml (0.2.5-r1)
#5 26.01 (10/10) Installing py3-yaml (6.0-r3)
#5 26.21 Executing busybox-1.36.1-r0.trigger
#5 26.24 OK: 57 MiB in 48 packages
#5 DONE 26.9s

#6 [3/3] RUN pip install pyyaml
#6 sha256:b98d84af5341c46e0664b7fe546494ec8d179b36b63c517b2bcf94a8f75d7b93
#6 9.049 Requirement already satisfied: pyyaml in /usr/lib/python3.11/site-packages (6.0)
#6 9.143 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#6 9.547 
#6 9.547 [notice] A new release of pip is available: 23.1.2 -> 23.2
#6 9.547 [notice] To update, run: pip install --upgrade pip
#6 DONE 10.1s

#7 exporting to image
#7 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#7 exporting layers
#7 exporting layers 1.8s done
#7 writing image sha256:1197cdcf014fbc6119475b76a13f04cd58f56fb8e263c4dbd7bda594246e60b6 done
#7 DONE 1.9s
millouzgc commented 11 months ago

ah thanks for the help, i was going crazy. for uvicorn project (py3.11), workaround it's ok

$ poetry add pyyaml==5.3.1

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/fd/01/723aae6192e3ac65338da311ea0bfe860ed243a951a96d8a936f3c3c7383/SQLAlchemy-2.0.19-py3-none-any.whl  91
Resolving dependencies... (3.6s)

Package operations: 0 installs, 1 update, 0 removals

  • Updating pyyaml (6.0 -> 5.3.1)

Writing lock file
galgertz commented 11 months ago

Another temporary workaround I used to install Pyyaml 5.4.1 was freezing the Cython version in pyproject.toml and installing Pyyaml from my forked git repo.

pyproject.toml

requires = ["setuptools", "wheel", "Cython==0.29.36"]

pip3 install git+https://github.com/galgertz/pyyaml.git@5.4.1_freeze_Cython

This is a similar solution to this PR: https://github.com/yaml/pyyaml/pull/702

azafred commented 11 months ago

Out of curiosity, any chance of https://github.com/yaml/pyyaml/pull/702 being merged in soon, or should everybody go ahead and implement local workarounds? There are quite a few projects relying on PyYaml to be working...

JoshWisniewski00 commented 11 months ago

Given the 5.3.1 work around has CVE: https://github.com/advisories/GHSA-8q59-q68h-6hv4 When will an updated release be available and what version do you anticipate it being?

timothyspangler commented 11 months ago

👀 👀 👀 👀

tschaffter commented 11 months ago

Thanks @olliemath! The following solution adapted from yours works for us:

        "commands": [
          "pyenv install --skip-existing 3.10.0",
          "pyenv local 3.10.0",
          "poetry env use 3.10.0",
          "poetry run pip install \"cython<3.0.0\"",
          "poetry run pip install --no-build-isolation pyyaml==5.4.1",
          "poetry install --with prod,dev"
        ],

EDIT: The option --no-build-isolation is needed.

greyli-dell commented 11 months ago

You can also upgrade to 6.0.1, which pins the Cython < 3.0.0.

S0b1t commented 11 months ago

You can also upgrade to 6.0.1, which pins the Cython < 3.0.0.

Confirm. Upgrading to 6.0.1 helped me too!

wzin commented 11 months ago

Freezing pyyaml to 5.3.1 and 6.0.1 solves the issue. I prefer 6.0.1.

grilo13 commented 11 months ago

I'm using keycloak version 3.1.3 that depends on pyyaml 5.4.1 so I'm not able to change to an older or newer version of it. Also I'm using poetry for the dependency management. Any idea on how to solve this temporarily?

Adityanagraj commented 11 months ago

Mostly in our case we removed pyyaml

augi commented 11 months ago

I'm using keycloak version 3.1.3 that depends on pyyaml 5.4.1 so I'm not able to change to an older or newer version of it. Also I'm using poetry for the dependency management. Any idea on how to solve this temporarily?

The same issue with docker-compose Python dependency.

johnthagen commented 11 months ago

7sxplp

luabida commented 11 months ago

Thanks @olliemath, your command saved the day. It also works with PyYaml < 6.0, and now I can at least move forward with the environment installation: pip install "cython<3.0.0" && pip install --no-build-isolation "pyyaml<6.0"

aliaksandrsauras commented 11 months ago

What about explicitly specifying working Cython version in the file pyproject.toml to avoid similar issue in future? For example: requires = ["setuptools", "wheel", "Cython==3.0.0"]

idanmiara commented 11 months ago

Thanks @luabida ! Should we backport https://github.com/yaml/pyyaml/pull/702 for PyYAML>=5.4,<6 for a more permanent workaround/fix?

henryiii commented 11 months ago

What about explicitly specifying working Cython

That's exactly what was done:

https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.toml

(FYI, you don't need "wheel" there)

Long term the fix is to fix the issue with Cython, as I'm sure people will want Cython 3 (and Cython 0.x will probably not support an upcoming version of Python if they don't back port fixes).

nitinsharmasf commented 11 months ago

You can use PyYaml 5.3.1 until the issue is resolved.

Please do not use this version. PyYAML version 5.3.1 is associated with CVE-2020-14343 that was fixed in version 5.4.

Instead use 6.0.1

chlndh commented 11 months ago

but docker-compose require PyYAML < 6. will there be a 5.4.2?

idanmiara commented 11 months ago

but docker-compose require PyYAML < 6. will there be a 5.4.2?

https://github.com/yaml/pyyaml/pull/726#issuecomment-1640411754

nitinsharmasf commented 11 months ago

but docker-compose require PyYAML < 6. will there be a 5.4.2?

See this: https://pypi.org/project/PyYAML/5.4/#history

xmnlab commented 11 months ago

but docker-compose require PyYAML < 6. will there be a 5.4.2?

that is one of the reasons we started to work on a rebundle of the docker-compose v2: https://pypi.org/project/compose-go/

cc @luabida

kay-bluhuntr commented 11 months ago

Was able to fix this by updating to the latest awscli v1(1.29.4) as it was a dependency for awscli. This pinned pyyaml to v 6.0.1

NeonDaniel commented 11 months ago

Not sure what I'm missing here, but I'm getting the same exceptions when explicitly installing cython<3.0.0 before pyyaml~=5.4 https://github.com/NeonGeckoCom/NeonCore/actions/runs/5590442924/jobs/10220174498

Thanks @olliemath, your command saved the day. It also works with PyYaml < 6.0, and now I can at least move forward with the environment installation: pip install "cython<3.0.0" && pip install --no-build-isolation "pyyaml<6.0"

Found this in the thread which worked to patch things, hopefully only temporarily.

MrMino commented 11 months ago

@NeonDaniel you're missing build isolation. Pip will not use your environment for building a wheel unless you explicitly tell it to use --no-build-isolation.

hellozdp commented 11 months ago

6.0.1 can support python3 user, but python2.7 support is removed in 6.0.0，so python2.7 user needs a fix in 5.4.x

berzi commented 11 months ago

A little recap, do correct me if I'm wrong:

If PyYAML is your own dependency or your dependencies support PyYAML~=6

If your problem is related to awscli

If your problem is related to aws-sam-cli or another package which requires PyYAML < 6

AlexeyMinasyan commented 11 months ago

@berzi Just a small comment on summary above - there is a typo in pyyaml version. Should 5.3.1 instead of 3.5.1

berzi commented 11 months ago

@AlexeyMinasyan Corrected, thank you.

mathieumalenfant commented 11 months ago

A little recap, do correct me if I'm wrong:

If PyYAML is your own dependency or your dependencies support PyYAML~=6

  • If you're on Python 3: bump PyYAML to at least 6.0.1.
  • If you need to support Python 2: use PyYAML 5.3.1 (NOT RECOMMENDED due to security issues. Consider updating Python.)

If your problem is related to awscli

  • If you can bump awscli to at least 1.29.4, do so.
  • If you can't, see the solution for aws-sam-cli below.

If your problem is related to aws-sam-cli or another package which requires PyYAML < 6

  • Before installing your other dependencies: pip install "cython<3.0.0" wheel && pip install pyyaml==5.4.1 --no-build-isolation
  • Some users report not needing wheel; that might be dependent on your preinstalled packages; on CI, I needed to include it.
  • If you're still getting errors (for missing commands during installation), try including setuptools alongside wheel.

Even If I use awscli 1.29.5, I still have the same issue...

berzi commented 11 months ago

@mathieumalenfant I suggest trying the next option then (the one for aws-sam-cli).

mathieumalenfant commented 11 months ago

@berzi All right, thanks. Hopefully they'll release a new awscli version soon that will fix the issue...

berzi commented 11 months ago

In my experience they tend to be far too strict with dependency versions. I recently had to correct a similar problem that prevented me from having both awscli and aws-sam-cli installed. Maybe it would be as simple as supporting PyYAML 6.0.1 for them.

realFranco commented 11 months ago

Hello @AlexDld ,

You can use PyYaml 5.3.1 until the issue is resolved.

This will introduce a vulnerability:

CVE-2020-14343

LouissXI commented 11 months ago

Hello @AlexDld ,

You can use PyYaml 5.3.1 until the issue is resolved.

This will introduce a vulnerability:

CVE-2020-14343

Hi @realFranco

Do you have any fix to this error without having the vulnerability ?

realFranco commented 11 months ago

Hello @LouissXI ,

Unfortunately no, I add it as a disclaimer and expose the consequences of install the package in that version.