Plone 6.0 on Python 3.11

[mauritsvanrees]

We now have the latest lxml, which allows a Python 3.11 buildout to finish. Jenkins still gives lots of errors, see console. There are even more errors than tests. :-) Summary:

Total: 491 tests, 4 failures, 768 errors and 2 skipped in 10.290 seconds.

Most errors are this:

  File "/home/jenkins/.buildout/eggs/cp311/persistent-4.9.0-py3.11-linux-x86_64.egg/persistent/ring.py", line 22, in <module>
    from persistent import _ring
ImportError: libffi.so.6: cannot open shared object file: No such file or directory

That sounds like we need to install a newer libffi on the nodes, or maybe persistent 4.9.1 will fix this, as its changelog says "Update Python 3.11 support to 3.11.0-rc1." It includes a wheel for 3.11, so that may help.

Locally, on my Mac, with older persistent 4.9.0, I can start an instance, create a Plone Site, edit a page, and run bin/test -u with only one failure in plone.memoize which indeed seems related to 3.11: AttributeError: module 'inspect' has no attribute 'getargspec'. bin/test -s borg.localrole passes, where this is the first one on Jenkins that shows the error for persistent. So it looks promising.

First step would be to create a PR job for 3.11, and then we can see if a newer persistent version helps. I will open an issue.

Anyway, this is low priority, we don't need this for a 6.0 final. But nice if it works. :-)

[mauritsvanrees]

Note that before Plone (or Zope) can officially support 3.11, more fixes and a security audit are needed for RestrictedPython.

[mauritsvanrees]

Together with @fredvd we have removed persistent and cffi from pip cache and buildout cache, and now it seems okay. Running now.

[fredvd]

While debugging the issue with Maurits I disabled the global pip cache on node1 to help with identifying the issue. I've created /home/jenkins/.config/pip/pip.conf with

cat ~/.config/pip/pip.conf
[global]
no-cache-dir = True

What we also tried was adding no-binary = :ALL: , to block pip install from using binary wheels, but this gives a very interesting traceback where buildout still tries to copy the expected binaries to produce an installed egg, but we block creating binaries. This is b.t.w. also documented in pip manual:

--no-binary

Do not use binary packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either >“:all:” to disable all binary packages, “:none:” to empty the set (notice the colons), or one or more package names with >commas between them (no colons). Note that some packages are tricky to compile and may fail to install when this >option is used on them.

---

In the end the offending broken dynamic link library , but it was the 'generic' cffi extension that was in the buildout cache which linked to libffi6. Which is very strange, AFAIK Node1 was upgraded to Ubuntu 22 some time ago. The timestamp on the .so below was from August 25th.

jenkins@node1:~/.buildout/eggs/cp311/cffi-1.15.0-py3.11-linux-x86_64.egg$ ldd _cffi_backend.cpython-311-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffd34251000)
    libffi.so.6 => not found
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1006fed000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f10069d8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1007000000)

[mauritsvanrees]

Okay, the pip conf change gave errors, seemingly already when creating a virtualenv, see the console log:

New python executable in /home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/bin/python3.9
Also creating executable in /home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /home/jenkins/shinin...d3bfc1/bin/python3.9 - setuptools pip wheel:
  Exception:
Traceback (most recent call last):
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/commands/install.py", line 241, in run
    with self._build_session(options) as session:
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/basecommand.py", line 77, in _build_session
    normalize_path(os.path.join(options.cache_dir, "http"))
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/lib/python3.9/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not int
Traceback (most recent call last):
  File "<stdin>", line 27, in <module>
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/__init__.py", line 246, in main
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/basecommand.py", line 261, in main
  File "/home/jenkins/shiningpanda/jobs/88954158/pip-0-py2.py3-none-any.whl/pip/_internal/basecommand.py", line 77, in _build_session
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/lib/python3.9/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not int

So we reverted this change. Should be fine again.

[mauritsvanrees]

Testing a 3.11 regex fix in PortalTransforms now.

[gforcada]

oh my :-/ the libffi error is probably because there was an existing package already built with it, and indeed, on Ubuntu 22.04 it is no longer there 👍🏾

btw. I think I will have some more time from now on, so ping me on discord or here. I'm starting to slowly come back to be more active 😄

[mauritsvanrees]

Plone 6.0 on Python 3.11 is green: https://jenkins.plone.org/job/plone-6.0-python-3.11/148/ There is no Jenkins job for robot tests yet though. I will open a separate issue in the Jenkins repo.