yaml / pyyaml

Canonical source repository for PyYAML
MIT License
2.47k stars 507 forks source link

" No module named '_ctypes'" when installing PyYAML 6.0.1 into Python 3.11.4 #742

Closed MartinBonner closed 10 months ago

MartinBonner commented 10 months ago

I have a freshly built copy of Python 3.11.4. I have installed Cython, pip, setuptools and wheel - but when I install PyYAML it fails.

Two things bother me about the following log:

/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-cxvu7wdg/yaml/_yaml.pxd

(I'm running python3) and

    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 28, in <module>
      from .macosx_libfile import calculate_macosx_platform_tag

(uname -a reports Linux 30e04d129175 5.15.0-78-generic #85~20.04.1-Ubuntu SMP Mon Jul 17 09:42:39 UTC 2023 x86_64 GNU/Linux)

Note that https://artifactory.ncipher.com/artifactory/api/pypi is a passthrough to pypi.org.

Other people seem to be able to install PyYAML 6.0.1, so what am I doing wrong?


# pip3 list
Package    Version
---------- -------
Cython     0.29.36
pip        23.1.2
setuptools 68.1.0
wheel      0.41.1
# pip3 install --verbose PyYAML-6.0.1.tar.gz
Using pip 23.1.2 from /usr/local/lib/python3.11/site-packages/pip (python 3.11)
Looking in indexes: https://artifactory.ncipher.com/artifactory/api/pypi/pypi-all/simple
Processing ./PyYAML-6.0.1.tar.gz
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://artifactory.ncipher.com/artifactory/api/pypi/pypi-all/simple
  Collecting setuptools
    Using cached https://artifactory.ncipher.com/artifactory/api/pypi/pypi-all/packages/packages/b8/49/b3b29c52b09075fb77f69309763a563b4054d5808a3f3b95df3a62ef3d3f/setuptools-68.1.0-py3-none-any.whl (805 kB)
  Collecting wheel
    Using cached https://artifactory.ncipher.com/artifactory/api/pypi/pypi-all/packages/packages/28/f5/6955d7b3a5d71ce6bac104f9cf98c1b0513ad656cdaca8ea7d579196f771/wheel-0.41.1-py3-none-any.whl (64 kB)
  Collecting Cython<3.0
    Using cached https://artifactory.ncipher.com/artifactory/api/pypi/pypi-all/packages/packages/3f/d6/9eed523aeaca42acbaa3e6d3850edae780dc7f8da9df1bf6a2ceb851839c/Cython-0.29.36-py2.py3-none-any.whl (988 kB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-0.29.36 setuptools-68.1.0 wheel-0.41.1
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing lib/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib/PyYAML.egg-info/top_level.txt
  cythoning yaml/_yaml.pyx to yaml/_yaml.c
  /tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-cxvu7wdg/yaml/_yaml.pxd
    tree = Parsing.p_module(s, pxd, full_module_name)
  reading manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'README'
  adding license file 'LICENSE'
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info
  writing /tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/SOURCES.txt'
  skipping 'yaml/_yaml.c' Cython extension (up-to-date)
  reading manifest file '/tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'README'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-nnssv7vl/PyYAML.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-nnssv7vl/PyYAML-6.0.1.dist-info'
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 396, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 289, in <module>
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1233, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/command/dist_info.py", line 112, in run
      bdist_wheel = self.get_finalized_command('bdist_wheel')
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
      cmd_obj = self.distribution.get_command_obj(command, create)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
      klass = self.get_command_class(command)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 980, in get_command_class
      self.cmdclass[command] = cmdclass = ep.load()
                                          ^^^^^^^^^
    File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
      module = import_module(match.group('module'))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 940, in exec_module
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 28, in <module>
      from .macosx_libfile import calculate_macosx_platform_tag
    File "/tmp/pip-build-env-0nqq5irb/overlay/lib/python3.11/site-packages/wheel/macosx_libfile.py", line 43, in <module>
      import ctypes
    File "/usr/local/lib/python3.11/ctypes/__init__.py", line 8, in <module>
      from _ctypes import Union, Structure, Array
  ModuleNotFoundError: No module named '_ctypes'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/bin/python3.11 /usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4d9f_96m
  cwd: /tmp/pip-req-build-cxvu7wdg
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[30e04d129175] root@docker:/home/mbonner/git/main/build-host-linux/external/python3/python3/additional-packages#`
MartinBonner commented 10 months ago

As a colleague has just got confused by this: 1. I am installing the tarball, not the wheel. 2. I am installing on Linux, not Mac.

kidrahahjo commented 10 months ago

Update Its related to https://github.com/yaml/pyyaml/issues/724 sorry for adding noise to this issue.

Hi I am facing an issue similar to this,

OS

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Python: 3.10.6

When I try to install "pyyaml<6", I get AttributeError: cython_sources

Existing packages in venv

(.venv)  ~/src/test  pip list
Package    Version
---------- -------
pip        23.2.1
setuptools 68.1.2

Traceback for the issue

(.venv) ~/src/test  pip install "pyyaml<6"
Collecting pyyaml<6
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 KB 4.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      /tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/hardikojha/src/test/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/hardikojha/src/test/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/hardikojha/src/test/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1233, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 319, in run
          self.find_sources()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 327, in find_sources          mm.run()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 549, in run
          self.add_defaults()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 587, in add_defaults          sdist.add_defaults(self)
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-gny7xrjs/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
ben9923 commented 10 months ago

You probably had a missing library when building Python so ctypes isn't available. Seems like you're using Ubuntu, try running sudo apt install libffi-dev, then re-building Python (make sure you re-run ./configure ...)

That said, wheel could probably be updated to work without ctypes being available in most of the flows, as it doesn't seem to always be necessary.

nitzmahone commented 10 months ago

Yeah, this definitely looks like a case of "you don't have a fully-functional Python installation"- Debian-based distros are notorious for applying their own modularization to Python installations, so it's possible to end up with parts of the Python standard library missing or broken without the installation of some optional packages.

Regardless, the errors you're hitting here have nothing to do with PyYAML- they're several layers beneath it in code we don't control, and if your Python's ctypes package isn't functional, your Python is going to be broken for many, many things.

Closing as "not our bug".

MartinBonner commented 10 months ago

Thanks! That is very helpful. Given that we do something a bit weird with libffi (so that the resulting Python is easy to install on lots of distros), it is entirely plausible that the problem is mine, but your diagnosis helps me track down what has changed between Python 3.10 (which worked) and Python 3.11 (which doesn't).

I agree "closing" is the right action.