search

aio-libs / yarl

Yet another URL library
https://yarl.aio-libs.org
Apache License 2.0
1.3k stars 160 forks source link

Building the wheel fails with TypeError: '_GeneratorContextManager' object is not an iterator #1013

Closed penguinpee closed 3 months ago

penguinpee commented 4 months ago

Describe the bug

While preparing to update yarl for Fedora to 1.9.4, the build failed with TypeError: '_GeneratorContextManager' object is not an iterator.

First, I encountered the infinite recursion described in #992. I applied the suggested fix from https://github.com/aio-libs/yarl/issues/992#issuecomment-1934720352 as a patch. After that I ran into the TypeError.

The issue is reproducible locally in a venv.

To Reproduce

  1. Checkout v1.9.4 and apply patch from https://github.com/aio-libs/yarl/issues/992#issuecomment-1934720352
  2. Setup venv
  3. Install required dependencies
  4. pip wheel --wheel-dir pyproject-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --verbose . (That's how the wheel is build in our build environment)

Expected behavior

The wheel is build.

Logs/tracebacks

$ pip wheel --wheel-dir pyproject-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --verbose .
Processing /home/sandro/devel/yarl
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info
  writing /tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*.cache' found anywhere in distribution
  warning: no previously-included files found matching 'yarl/*.c'
  warning: no previously-included files found matching 'yarl/*.html'
  warning: no previously-included files found matching 'yarl/*.so'
  warning: no previously-included files found matching 'yarl/*.pyd'
  no previously-included directories found matching 'docs/_build'
  adding license file 'LICENSE'
  adding license file 'NOTICE'
  writing manifest file '/tmp/pip-modern-metadata-bgfp4kul/yarl.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-bgfp4kul/yarl-1.9.4.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: yarl
  Running command Building wheel for yarl (pyproject.toml)
  **********************
  * Accelerated build *
  **********************
  Traceback (most recent call last):
    File "/home/sandro/devel/yarl/wheel/lib64/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/sandro/devel/yarl/wheel/lib64/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/sandro/devel/yarl/wheel/lib64/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib64/python3.12/contextlib.py", line 81, in inner
      return func(*args, **kwds)
             ^^^^^^^^^^^^^^^^^^^
    File "/home/sandro/devel/yarl/packaging/pep517_backend/_backend.py", line 297, in build_wheel
      with maybe_prebuild_c_extensions(
    File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__
      return next(self.gen)
             ^^^^^^^^^^^^^^
    File "/home/sandro/devel/yarl/packaging/pep517_backend/_backend.py", line 271, in maybe_prebuild_c_extensions
      with build_dir_ctx:
    File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__
      return next(self.gen)
             ^^^^^^^^^^^^^^
  TypeError: '_GeneratorContextManager' object is not an iterator
  error: subprocess-exited-with-error

  × Building wheel for yarl (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/sandro/devel/yarl/wheel/bin/python3 /home/sandro/devel/yarl/wheel/lib64/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmp9hpu3tqo
  cwd: /home/sandro/devel/yarl
  Building wheel for yarl (pyproject.toml) ... error
  ERROR: Failed building wheel for yarl
Failed to build yarl
ERROR: Failed to build one or more wheels

Python Version

$ python --version
Python 3.12.3

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/lib64/python3.12/site-packages
Requires: 
Required-by: aiohttp, httpie, yarl

yarl Version

$ python -m pip show yarl
WARNING: Package(s) not found: yarl ;)

OS

NAME="Fedora Linux" VERSION="39 (Thirty Nine)" ID=fedora VERSION_ID=39 VERSION_CODENAME="" PLATFORM_ID="platform:f39" PRETTY_NAME="Fedora Linux 39 (Thirty Nine)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:39" DEFAULT_HOSTNAME="fedora" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=39 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=39 SUPPORT_END=2024-11-12

Additional context

Build dependencies are coming from Fedora repos. So, versions may differ to what pip would install. Above information is with regards to the local reproducer.

Notably, when passing -C pure-python=1 to pip, the wheel is built successfully. But since we want the C extensions, that's not a way out.

I also saw #1001. But this looks different.

Code of Conduct

webknjaz commented 3 months ago

@penguinpee it's likely that you misapplied that patch and lost yield along the way. A missing yield under @contextmanager is the only reasonable explanation. I'm sure if you git diff your thing and look into it carefully, you'll spot what's gone wrong.

penguinpee commented 3 months ago

Hmm. I'm stupefied. I just ran another build and it works. I can not recall what I did differently two weeks ago. Anyway, it wasn't the patch applied. That looks sane (except for the added @contextmanager):

diff --git a/packaging/pep517_backend/_backend.py b/packaging/pep517_backend/_backend.py
index 9a28ace..e09fd06 100644
--- a/packaging/pep517_backend/_backend.py
+++ b/packaging/pep517_backend/_backend.py
@@ -194,12 +194,22 @@ def patched_dist_get_long_description():
         _DistutilsDistributionMetadata.get_long_description = _orig_func

+@contextmanager
 @contextmanager
 def _in_temporary_directory(src_dir: Path) -> t.Iterator[None]:
     with TemporaryDirectory(prefix='.tmp-yarl-pep517-') as tmp_dir:
         with chdir_cm(tmp_dir):
             tmp_src_dir = Path(tmp_dir) / 'src'
-            copytree(src_dir, tmp_src_dir, symlinks=True)
+            def _ignore(d, _):
+                """
+                Prevent temporary directory to be copied into self
+                recursively forever.
+                Fixes https://github.com/aio-libs/yarl/issues/992
+                """
+                if Path(d) == tmp_src_dir.parent:
+                    return [tmp_src_dir.name]
+                return []
+            copytree(src_dir, tmp_src_dir, symlinks=True, ignore=_ignore)
             os.chdir(tmp_src_dir)
             yield

Well, I can proceed. I won't wack my :brain: about it.

webknjaz commented 3 months ago

That looks sane (except for the added @contextmanager):

Yep, double @contextmanager would cause that.

penguinpee commented 3 months ago

I see. That would explain it. My :brain: woke up and I know now why it suddenly worked. I changed the build config to build a pure Python package just as a trial.

I had to revert all that. But with the fixed patch, it now builds with C extensions as well. All good! :tada: