pip install fails on Python 3.11b3

[oscarbenjamin]

This is from SymPy's CI: https://github.com/sympy/sympy/runs/6711439310?check_suite_focus=true

You can see the build steps here: https://github.com/sympy/sympy/blob/ff00553cf2a03ce048bf46c25c8a3cbe2a636c2c/.github/workflows/runtests.yml#L186-L200

I'm not sure which dependency pulls in kiwisolver but pip fails to build it on Ubuntu with Python 3.11b3 from these steps:

      # Install the non-Python dependencies
      sudo apt install antlr4 libgfortran5 gfortran libmpfr-dev libmpc-dev libopenblas-dev
      python -m pip install --upgrade pip wheel

      pip install git+https://github.com/cython/cython@master
      pip install git+https://github.com/numpy/numpy@main
      pip install git+https://github.com/scipy/scipy@main

      # dependencies to install in all Python versions:
      pip install mpmath matplotlib numpy ipython cython scipy aesara \
                         wurlitzer autowrap numexpr 'antlr4-python3-runtime==4.7.*'

This was all working a week or so ago with Python 3.11b1 so I'm not sure what has changed.

The relevant part of the error message:

  Building wheel for kiwisolver (pyproject.toml): started
  Building wheel for kiwisolver (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Building wheel for kiwisolver (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      /tmp/pip-build-env-6pl9bobj/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:102: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
        warnings.warn(msg, _ExperimentalProjectMetadata)
      /tmp/pip-build-env-6pl9bobj/overlay/lib/python3.11/site-packages/setuptools/config/_apply_pyprojecttoml.py:101: _WouldIgnoreField: 'dependencies' defined outside of `pyproject.toml` would be ignored.
          !!

          ##########################################################################
          # configuration would be ignored/result in error due to `pyproject.toml` #
          ##########################################################################

          The following seems to be defined outside of `pyproject.toml`:

          `dependencies = ["typing_extensions;python_version<'3.8'"]`

          According to the spec (see the link bellow), however, setuptools CANNOT
          consider this value unless 'dependencies' is listed as `dynamic`.

  Building wheel for pillow (setup.py): started
          https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

          For the time being, `setuptools` will still consider the given value (as a
          **transitional** measure), but please note that future releases of setuptools will
          follow strictly the standard.

          To prevent this warning, you can list 'dependencies' under `dynamic` or alternatively
          remove the `[project]` table from your file and rely entirely on other means of
          configuration.

      !!

        warnings.warn(msg, _WouldIgnoreField)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/kiwisolver
      copying py/kiwisolver/__init__.py -> build/lib.linux-x86_64-cpython-311/kiwisolver
      copying py/kiwisolver/py.typed -> build/lib.linux-x86_64-cpython-311/kiwisolver
      copying py/kiwisolver/_cext.pyi -> build/lib.linux-x86_64-cpython-311/kiwisolver
      running build_ext
      building 'kiwisolver._cext' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/py
      creating build/temp.linux-x86_64-cpython-311/py/src
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-build-env-6pl9bobj/overlay/lib/python3.11/site-packages/cppy/include -I. -I/opt/hostedtoolcache/Python/3.11.0-beta.3/x64/include/python3.11 -c py/src/constraint.cpp -o build/temp.linux-x86_64-cpython-311/py/src/constraint.o -std=c++11
      In file included from /opt/hostedtoolcache/Python/3.11.0-beta.3/x64/include/python3.11/Python.h:38,
                       from /tmp/pip-build-env-6pl9bobj/overlay/lib/python3.11/site-packages/cppy/include/cppy/defines.h:10,
                       from /tmp/pip-build-env-6pl9bobj/overlay/lib/python3.11/site-packages/cppy/include/cppy/cppy.h:16,
                       from py/src/constraint.cpp:10:
      /opt/hostedtoolcache/Python/3.11.0-beta.3/x64/include/python3.11/pyport.h: In instantiation of ‘type {anonymous}::_Py_CAST_impl(const expr_type&) [with type = _object*; expr_type = int]’:
      py/src/util.h:120:9:   required from here
      /opt/hostedtoolcache/Python/3.11.0-beta.3/x64/include/python3.11/pyport.h:47:24: error: invalid static_cast from type ‘int’ to type ‘_object*’
         47 |                 return static_cast<type>(const_cast<expr_type &>(expr));
            |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for kiwisolver

[MatthieuDartiailh]

Thanks for the report. I did not yet start testing with 3.11. I hope to get to it soon though.

[nascheme]

This seems to be an issue with type casts with the new inline functions included in 3.11.x. I've created a CPython bug since I don't know if it is a bug with CPython or a bug with the kiwisolver code. See:

https://github.com/python/cpython/issues/93442

Using PyTuple_SetItem fixes the issue but I don't know if the new PyTuple_SET_ITEM is buggy or if the kiwi C++ source code is doing something not quite correct. I'm not a C++ expert.

[nascheme]

The issue can be worked around by using nullptr, as per change below. However, I think the PyTuple_SET_ITEM() and similar macros in CPython 3.11 should be changed to accept NULL/0 as null arguments.

diff --git a/py/src/symbolics.h b/py/src/symbolics.h
index 69ea540..1558063 100644
--- a/py/src/symbolics.h
+++ b/py/src/symbolics.h
@@ -123,7 +123,7 @@ PyObject* BinaryMul::operator()( Expression* first, double second )
                return 0;
        Py_ssize_t end = PyTuple_GET_SIZE( first->terms );
        for( Py_ssize_t i = 0; i < end; ++i )  // memset 0 for safe error return
-               PyTuple_SET_ITEM( terms.get(), i, 0 );
+               PyTuple_SET_ITEM( terms.get(), i, nullptr );
        for( Py_ssize_t i = 0; i < end; ++i )
        {
                PyObject* item = PyTuple_GET_ITEM( first->terms, i );
diff --git a/py/src/util.h b/py/src/util.h
index 4b00fa7..1f40dc9 100644
--- a/py/src/util.h
+++ b/py/src/util.h
@@ -117,7 +117,7 @@ make_terms( const std::map<PyObject*, double>& coeffs )
         return 0;
     Py_ssize_t size = PyTuple_GET_SIZE( terms.get() );
     for( Py_ssize_t i = 0; i < size; ++i ) // zero tuple for safe early return
-        PyTuple_SET_ITEM( terms.get(), i, 0 );
+        PyTuple_SET_ITEM( terms.get(), i, nullptr );
     Py_ssize_t i = 0;
     iter_t it = coeffs.begin();
     iter_t end = coeffs.end();

[MatthieuDartiailh]

Thanks for investigating. I will make a PR with those changes while waiting to see if a fix lands in CPython and add 3.11 to the test matrix.

[MatthieuDartiailh]

PR is at #142 and appears to work just fine. I will merge sometimes next week if nothing happens in the meantime.

[oscarbenjamin]

Thanks!

[MatthieuDartiailh]

The release process has started but the aarch build is real slow. Hopefully the wheels will appear on PyPI later today

[neutrinoceros]

The build is much longer than the typical duration I can see in the history. Maybe it's just randomly hanging ?

[MatthieuDartiailh]

Right I will trig it again

[MatthieuDartiailh]

The discussed fix appears to fail on MacOS ARM (see #144). I would appreciate if anybody with more C++ background has an idea on how to fix this for all compilers and architectures.

[neutrinoceros]

I think the following CPython patch may have made #142 unecessary : https://github.com/python/cpython/commit/8bcc3fa3453e28511d04eaa0aa7d8e1a3495d518

This patch was not yet shipped, and will (likely) be included only in Python 3.11.0b4 (scheduled for this Thursday). I hope this helps.

[MatthieuDartiailh]

Thanks I will look into reverting the change once the beta is out.

[neutrinoceros]

@MatthieuDartiailh FYI Python 3.11.0b4 is now available after a couple weeks of delay :)

[MatthieuDartiailh]

Thanks I am waiting for it to be available on GHA to merge the relevant PR.

[neutrinoceros]

It's been available on GHA for a couple hours

[oscarbenjamin]

Everything seems to be installing fine for SymPy with 3.11b4: https://github.com/sympy/sympy/runs/7318381460

There is a test failure but it's unrelated to kiwi.

[MatthieuDartiailh]

I merged the PR and will try to cut a new release soon.