django_template and sympy failing because of distils dependency

[diegorusso]

As the title says, these two benchmarks are failing because they need distutils

[2024-06-27T23:40:53.242Z] Traceback (most recent call last):
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/lib/python3.14/site-packages/pyperformance/data-files/benchmarks/bm_sympy/run_benchmark.py", line 3, in <module>
[2024-06-27T23:40:53.242Z]     from sympy import expand, symbols, integrate, tan, summation
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/__init__.py", line 51, in <module>
[2024-06-27T23:40:53.242Z]     from .core import (sympify, SympifyError, cacheit, Basic, Atom,
[2024-06-27T23:40:53.242Z]     ...<11 lines>...
[2024-06-27T23:40:53.242Z]             Catalan, EulerGamma, GoldenRatio, TribonacciConstant)
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/core/__init__.py", line 4, in <module>
[2024-06-27T23:40:53.242Z]     from .sympify import sympify, SympifyError
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/core/sympify.py", line 9, in <module>
[2024-06-27T23:40:53.242Z]     from .compatibility import iterable
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/core/compatibility.py", line 11, in <module>
[2024-06-27T23:40:53.242Z]     from sympy.external import import_module
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/external/__init__.py", line 18, in <module>
[2024-06-27T23:40:53.242Z]     from sympy.external.importtools import import_module
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/cpython3.14-7b52106d442c-compat-2d3356be745c/lib/python3.14/site-packages/sympy/external/importtools.py", line 4, in <module>
[2024-06-27T23:40:53.242Z]     from distutils.version import LooseVersion
[2024-06-27T23:40:53.242Z] ModuleNotFoundError: No module named 'distutils'
[2024-06-27T23:40:53.242Z] Traceback (most recent call last):
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/lib/python3.14/site-packages/pyperformance/run.py", line 145, in run_benchmarks
[2024-06-27T23:40:53.242Z]     result = bench.run(
[2024-06-27T23:40:53.242Z]         bench_venv.python,
[2024-06-27T23:40:53.242Z]     ...<3 lines>...
[2024-06-27T23:40:53.242Z]         verbose=options.verbose,
[2024-06-27T23:40:53.242Z]     )
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 189, in run
[2024-06-27T23:40:53.242Z]     bench = _run_perf_script(
[2024-06-27T23:40:53.242Z]         python,
[2024-06-27T23:40:53.242Z]     ...<4 lines>...
[2024-06-27T23:40:53.242Z]         verbose=verbose,
[2024-06-27T23:40:53.242Z]     )
[2024-06-27T23:40:53.242Z]   File "/Users/bot/workspace/workspace/cpython/build-base/venv/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 236, in _run_perf_script
[2024-06-27T23:40:53.242Z]     raise RuntimeError("Benchmark died")
[2024-06-27T23:40:53.242Z] RuntimeError: Benchmark died
[2024-06-27T23:40:53.242Z] Command failed with exit code 1
[2024-06-27T23:40:53.242Z] ERROR: Benchmark sympy failed: Benchmark died

In the past I saw this dependency was added and removed. Maybe @mdboom might shed some lights :) Do we need it back? Can we update the version of django (3.2.4) or sympy (1.8) used in the benchmark?

I'm testing this with Cpython main branch but in theory should fail with 3.12 onwards because of the removal of distutils from the standard library.

[mdboom]

The reason the benchmarks are expected to work without it is that pyperformance always installs setuptools into the virtual environment in which the benchmarks run. These benchmarks are running on the latest weekly run on Linux ARM, for example: https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20240622-3.14.0a0-4717aaa/bm-20240622-arminc-aarch64-python-4717aaa1a72d1964f153-3.14.0a0-4717aaa-vs-3.10.4.md

The reason setuptools was removed as an explicit dependency for these benchmarks is that the version that was specified broke against CPython main, and it was easier to track a single version in pyperformance at the top level.

So the question is why is it working on the benchmark infrastructure and not for you? Are you running the benchmarks directly outside of pyperformance? Are there any clues in the part of the log above where the venvs are created?

[diegorusso]

This is odd indeed. Looking at the logs of my build I see.

[2024-06-28T09:07:50.142Z] (32/84) creating venv for benchmark (django_template)
[2024-06-28T09:07:50.142Z] 
[2024-06-28T09:07:50.142Z] (trying common venv first)
[2024-06-28T09:07:50.142Z] Installing requirements into the virtual environment /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c
[2024-06-28T09:07:50.142Z] # /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/bin/python -m pip install asgiref==3.3.4 django==3.2.4 pytz==2021.1 sqlparse==0.4.1 pyperf==2.6.1
[2024-06-28T09:07:50.846Z] Collecting asgiref==3.3.4
[2024-06-28T09:07:50.846Z]   Downloading asgiref-3.3.4-py3-none-any.whl.metadata (9.1 kB)
[2024-06-28T09:07:51.181Z] Collecting django==3.2.4
[2024-06-28T09:07:51.181Z]   Downloading Django-3.2.4-py3-none-any.whl.metadata (3.9 kB)
[2024-06-28T09:07:51.181Z] Collecting pytz==2021.1
[2024-06-28T09:07:51.181Z]   Downloading pytz-2021.1-py2.py3-none-any.whl.metadata (21 kB)
[2024-06-28T09:07:51.181Z] Collecting sqlparse==0.4.1
[2024-06-28T09:07:51.181Z]   Downloading sqlparse-0.4.1-py3-none-any.whl.metadata (3.4 kB)
[2024-06-28T09:07:51.181Z] Requirement already satisfied: pyperf==2.6.1 in ./venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/lib/python3.14/site-packages (2.6.1)
[2024-06-28T09:07:51.181Z] Requirement already satisfied: psutil>=5.9.0 in ./venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/lib/python3.14/site-packages (from pyperf==2.6.1) (6.0.0)
[2024-06-28T09:07:51.181Z] Downloading asgiref-3.3.4-py3-none-any.whl (22 kB)
[2024-06-28T09:07:51.181Z] Downloading Django-3.2.4-py3-none-any.whl (7.9 MB)
[2024-06-28T09:07:51.182Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.9/7.9 MB 55.9 MB/s eta 0:00:00
[2024-06-28T09:07:51.182Z] Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
[2024-06-28T09:07:51.182Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 510.8/510.8 kB 61.2 MB/s eta 0:00:00
[2024-06-28T09:07:51.182Z] Downloading sqlparse-0.4.1-py3-none-any.whl (42 kB)
[2024-06-28T09:07:51.182Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.2/42.2 kB 17.5 MB/s eta 0:00:00
[2024-06-28T09:07:51.516Z] Installing collected packages: pytz, sqlparse, asgiref, django
[2024-06-28T09:07:53.788Z] Successfully installed asgiref-3.3.4 django-3.2.4 pytz-2021.1 sqlparse-0.4.1
[2024-06-28T09:07:53.799Z] Installing requirements into the virtual environment /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c
[2024-06-28T09:07:53.799Z] # /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/bin/python -m pip install psutil
[2024-06-28T09:07:54.123Z] Requirement already satisfied: psutil in ./venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/lib/python3.14/site-packages (6.0.0)
[2024-06-28T09:07:54.457Z] # /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/bin/python -m pip freeze
[2024-06-28T09:07:54.792Z] asgiref==3.3.4
[2024-06-28T09:07:54.792Z] Chameleon==3.9.1
[2024-06-28T09:07:54.792Z] click==8.1.7
[2024-06-28T09:07:54.792Z] cloudpickle==3.0.0
[2024-06-28T09:07:54.792Z] coverage==7.3.2
[2024-06-28T09:07:54.792Z] dask==2022.2.0
[2024-06-28T09:07:54.792Z] distributed==2022.2.0
[2024-06-28T09:07:54.792Z] Django==3.2.4
[2024-06-28T09:07:54.792Z] fsspec==2024.6.1
[2024-06-28T09:07:54.792Z] Jinja2==3.1.4
[2024-06-28T09:07:54.792Z] locket==1.0.0
[2024-06-28T09:07:54.792Z] MarkupSafe==2.1.5
[2024-06-28T09:07:54.792Z] msgpack==1.0.8
[2024-06-28T09:07:54.792Z] packaging==24.1
[2024-06-28T09:07:54.792Z] partd==1.4.2
[2024-06-28T09:07:54.792Z] psutil==6.0.0
[2024-06-28T09:07:54.792Z] pyaes==1.6.1
[2024-06-28T09:07:54.792Z] pyperf==2.6.1
[2024-06-28T09:07:54.792Z] pytz==2021.1
[2024-06-28T09:07:54.792Z] PyYAML==6.0.1
[2024-06-28T09:07:54.792Z] setuptools==70.1.1         <=============== it is installed!!!
[2024-06-28T09:07:54.792Z] sortedcontainers==2.4.0
[2024-06-28T09:07:54.792Z] sqlparse==0.4.1
[2024-06-28T09:07:54.792Z] tblib==3.0.0
[2024-06-28T09:07:54.792Z] toolz==0.12.1
[2024-06-28T09:07:54.792Z] tornado==6.4.1
[2024-06-28T09:07:54.792Z] websockets==11.0.3
[2024-06-28T09:07:54.792Z] wheel==0.43.0
[2024-06-28T09:07:54.792Z] zict==3.0.0

but then when it tries to run it cannot find distutils

[2024-06-28T09:37:42.193Z] # /home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/bin/python -u /home/ent-user/ci-scripts/cpython/build-head/venv/lib/python3.14/site-packages/pyperformance/data-files/benchmarks/bm_django_template/run_benchmark.py --output /tmp/tmp0soiecdl --inherit-environ PYPERFORMANCE_RUNID
[2024-06-28T09:37:42.193Z] Traceback (most recent call last):
[2024-06-28T09:37:42.193Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/lib/python3.14/site-packages/pyperformance/data-files/benchmarks/bm_django_template/run_benchmark.py", line 8, in <module>
[2024-06-28T09:37:42.193Z]     import django.conf
[2024-06-28T09:37:42.193Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/lib/python3.14/site-packages/django/__init__.py", line 1, in <module>
[2024-06-28T09:37:42.193Z]     from django.utils.version import get_version
[2024-06-28T09:37:42.193Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/cpython3.14-bdb8f71d1bf2-compat-2d3356be745c/lib/python3.14/site-packages/django/utils/version.py", line 6, in <module>
[2024-06-28T09:37:42.193Z]     from distutils.version import LooseVersion
[2024-06-28T09:37:42.193Z] ModuleNotFoundError: No module named 'distutils'
[2024-06-28T09:37:42.193Z] Traceback (most recent call last):
[2024-06-28T09:37:42.193Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/lib/python3.14/site-packages/pyperformance/run.py", line 145, in run_benchmarks
[2024-06-28T09:37:42.193Z]     result = bench.run(
[2024-06-28T09:37:42.194Z]         bench_venv.python,
[2024-06-28T09:37:42.194Z]     ...<3 lines>...
[2024-06-28T09:37:42.194Z]         verbose=options.verbose,
[2024-06-28T09:37:42.194Z]     )
[2024-06-28T09:37:42.194Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 189, in run
[2024-06-28T09:37:42.194Z]     bench = _run_perf_script(
[2024-06-28T09:37:42.194Z]         python,
[2024-06-28T09:37:42.194Z]     ...<4 lines>...
[2024-06-28T09:37:42.194Z]         verbose=verbose,
[2024-06-28T09:37:42.194Z]     )
[2024-06-28T09:37:42.194Z]   File "/home/ent-user/ci-scripts/cpython/build-head/venv/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 236, in _run_perf_script
[2024-06-28T09:37:42.194Z]     raise RuntimeError("Benchmark died")
[2024-06-28T09:37:42.194Z] RuntimeError: Benchmark died
[2024-06-28T09:37:42.194Z] Command failed with exit code 1
[2024-06-28T09:37:42.194Z] ERROR: Benchmark django_template failed: Benchmark died

[diegorusso]

ok, I know what the problem is. I'm calling python from the build directory and this messes up with the import system.

(cpython3.14-1f5808d4c3c0-compat-2d3356be745c) dierus01 at ubuntu-vm in ~/repos/ci-scripts/projects/python/cpython/build-head on ci_test [?]$ python -m setuptools
Traceback (most recent call last):
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/Lib/runpy.py", line 189, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/Lib/runpy.py", line 148, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/Lib/runpy.py", line 112, in _get_module_details
    __import__(pkg_name)
    ~~~~~~~~~~^^^^^^^^^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/build-head/venv/cpython3.14-1f5808d4c3c0-compat-2d3356be745c/lib/python3.14/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/build-head/venv/cpython3.14-1f5808d4c3c0-compat-2d3356be745c/lib/python3.14/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/build-head/venv/cpython3.14-1f5808d4c3c0-compat-2d3356be745c/lib/python3.14/site-packages/_distutils_hack/__init__.py", line 70, in do_override
    ensure_local_distutils()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/build-head/venv/cpython3.14-1f5808d4c3c0-compat-2d3356be745c/lib/python3.14/site-packages/_distutils_hack/__init__.py", line 53, in ensure_local_distutils
    importlib.import_module('distutils')
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils'
(cpython3.14-1f5808d4c3c0-compat-2d3356be745c) dierus01 at ubuntu-vm in ~/repos/ci-scripts/projects/python/cpython/build-head on ci_test [?]$ cd ..
(cpython3.14-1f5808d4c3c0-compat-2d3356be745c) dierus01 at ubuntu-vm in ~/repos/ci-scripts/projects/python/cpython on ci_test [?]$ python -m setuptools
/home/dierus01/work/ce-sw/repos/ci-scripts/projects/python/cpython/build-head/venv/cpython3.14-1f5808d4c3c0-compat-2d3356be745c/bin/python: No module named setuptools.__main__; 'setuptools' is a package and cannot be directly executed

[diegorusso]

Now I have another import problem with this benchmark

[2024-06-28T16:45:04.378Z] [1/1] django_template...
[2024-06-28T16:45:04.378Z] # /private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/bin/python -u /private/tmp/venv-build-head/lib/python3.14/site-packages/pyperformance/data-files/benchmarks/bm_django_template/run_benchmark.py --output /var/folders/qd/2rph17h90ys0n1wp9pq99ccm0000z9/T/tmppmop67zn --inherit-environ PYPERFORMANCE_RUNID
[2024-06-28T16:45:04.378Z] Traceback (most recent call last):
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv-build-head/lib/python3.14/site-packages/pyperformance/data-files/benchmarks/bm_django_template/run_benchmark.py", line 37, in <module>
[2024-06-28T16:45:04.378Z]     django.setup()
[2024-06-28T16:45:04.378Z]     ~~~~~~~~~~~~^^
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/__init__.py", line 16, in setup
[2024-06-28T16:45:04.378Z]     from django.urls import set_script_prefix
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/urls/__init__.py", line 1, in <module>
[2024-06-28T16:45:04.378Z]     from .base import (
[2024-06-28T16:45:04.378Z]     ...<3 lines>...
[2024-06-28T16:45:04.378Z]     )
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/urls/base.py", line 8, in <module>
[2024-06-28T16:45:04.378Z]     from .exceptions import NoReverseMatch, Resolver404
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/urls/exceptions.py", line 1, in <module>
[2024-06-28T16:45:04.378Z]     from django.http import Http404
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/http/__init__.py", line 2, in <module>
[2024-06-28T16:45:04.378Z]     from django.http.request import (
[2024-06-28T16:45:04.378Z]         HttpRequest, QueryDict, RawPostDataException, UnreadablePostError,
[2024-06-28T16:45:04.378Z]     )
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv/cpython3.14-96cd427c1827-compat-2d3356be745c/lib/python3.14/site-packages/django/http/request.py", line 1, in <module>
[2024-06-28T16:45:04.378Z]     import cgi
[2024-06-28T16:45:04.378Z] ModuleNotFoundError: No module named 'cgi'
[2024-06-28T16:45:04.378Z] Traceback (most recent call last):
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv-build-head/lib/python3.14/site-packages/pyperformance/run.py", line 145, in run_benchmarks
[2024-06-28T16:45:04.378Z]     result = bench.run(
[2024-06-28T16:45:04.378Z]         bench_venv.python,
[2024-06-28T16:45:04.378Z]     ...<3 lines>...
[2024-06-28T16:45:04.378Z]         verbose=options.verbose,
[2024-06-28T16:45:04.378Z]     )
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv-build-head/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 189, in run
[2024-06-28T16:45:04.378Z]     bench = _run_perf_script(
[2024-06-28T16:45:04.378Z]         python,
[2024-06-28T16:45:04.378Z]     ...<4 lines>...
[2024-06-28T16:45:04.378Z]         verbose=verbose,
[2024-06-28T16:45:04.378Z]     )
[2024-06-28T16:45:04.378Z]   File "/private/tmp/venv-build-head/lib/python3.14/site-packages/pyperformance/_benchmark.py", line 236, in _run_perf_script
[2024-06-28T16:45:04.378Z]     raise RuntimeError("Benchmark died")

[mdboom]

The django_template benchmark does have a requirement on legacy-cgi to provide the cgi module on recent Pythons. Did that maybe fail to install for some other reason?

[diegorusso]

it does it indeed, version 1.11.0 :) I had it pinned to 1.10.0 hence it wasn't installing it.