sagemath_environment must respect --enable-system-site-packages

dimpase commented 4 months ago

Steps To Reproduce

Trigger building sagemath_environment, e.g. (after applying https://github.com/sagemath/sage/pull/38163) by make sagemath_tdlib, or from scratch, by

./bootstrap && ./configure --enable-system-site-packages --enable-sagemath_tdlib && make build

then, assuming that either setuptools or wheel packages come from the system, it will fail, as observed on https://github.com/sagemath/sage/issues/38159#issuecomment-2158657293

Expected Behavior

should work

Actual Behavior

in does not.

Additional Information

There is a lookup happening for wheels for setuptools and wheel - something that is almost sure to fail if the latter come from the system.

Environment

- **OS**: Gentoo Linux
- **10.4.beta9+#38163**:

Checklist

[X] I have searched the existing issues for a bug report that matches the one I want to file, without success.
[X] I have read the documentation and troubleshoot guide

dimpase commented 4 months ago

@orlitzky @kiwifb

saraedum commented 4 months ago

@dimpase am I correct that this is only a blocker if #38163 gets merged?

dimpase commented 4 months ago

it is a blocker if any use is made of sagemath_environment in its current state.

dimpase commented 4 months ago

more precisely it could be used with another experimental package, but not with optional or standard.

dimpase commented 4 months ago

As you see, this bug (two bugs in one - dependency on an exterimental package + breaking --enable-system-side-packages) has proliferated itself quite a bit

$ git grep sagemath_environment build/pkgs/
build/pkgs/sagemath_bliss/dependencies: bliss cysignals | $(PYTHON_TOOLCHAIN) sage_setup sage_conf sagemath_environment cython pkgconfig $(PYTHON)
build/pkgs/sagemath_categories/dependencies: sagemath_objects | $(PYTHON_TOOLCHAIN) sagemath_environment sage_setup cython pkgconfig python_build $(PYTHON)
build/pkgs/sagemath_coxeter3/dependencies: coxeter3 | $(PYTHON_TOOLCHAIN) sage_setup sagemath_environment cython pkgconfig $(PYTHON)
build/pkgs/sagemath_environment/spkg-src:cd build/pkgs/sagemath_environment
build/pkgs/sagemath_meataxe/dependencies: meataxe cysignals | $(PYTHON_TOOLCHAIN) sage_setup sagemath_environment cython pkgconfig $(PYTHON)
build/pkgs/sagemath_objects/dependencies:FORCE  cysignals gmpy2 | $(PYTHON_TOOLCHAIN) sagemath_environment sage_setup cython pkgconfig python_build $(PYTHON)
build/pkgs/sagemath_repl/dependencies: sagemath_objects sagemath_environment ipython ipywidgets | $(PYTHON_TOOLCHAIN) python_build $(PYTHON)
build/pkgs/sagemath_sirocco/dependencies: sirocco cypari cysignals mpfr | $(PYTHON_TOOLCHAIN) sage_setup sagemath_environment cython pkgconfig $(PYTHON)
build/pkgs/sagemath_tdlib/dependencies: tdlib cysignals boost_cropped | $(PYTHON_TOOLCHAIN) sage_setup sagemath_environment cython pkgconfig $(PYTHON)

of these, sagemath_objects, sagemath_repl, sagemath_categories are experimental. So these don't violate anything, they are just broken, wrt to the 2nd bug.

For the other ones, something needs to be done. The most natural is to fix sagemath_environment so that the 2nd bug is fixed, and promote it to optional, removing the 1st bug.

dimpase commented 4 months ago

I misread the diff for https://github.com/sagemath/sage/pull/38163 - the breaking change is much less recent. This certainly does not make the situation any better, just shifts the blame from one PR to another PR, #35661

dimpase commented 4 months ago

@dimpase am I correct that this is only a blocker if #38163 gets merged?

no, this is not correct, apologies. The breaking changes came from #35661 - @kiwifb : you didn't notice there that it made a number of optional packages dependent upon an experimental package, sagemath_environment.

And it has broken --enable-system-side-packages

dimpase commented 3 months ago

with 10.4.rc1, the issue is still here:

[sagemath_environment-10.4.rc1] Setting up build directory /mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/var/tmp/sage/build/sagemath_environment-10.4.rc1
[sagemath_environment-10.4.rc1] Host system: Linux hilbert 6.0.8-gentoo-x86_64 #2 SMP PREEMPT_DYNAMIC Sun Dec 18 22:31:12 GMT 2022 x86_64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz GenuineIntel GNU/Linux
[sagemath_environment-10.4.rc1] C compiler: gcc, Using built-in specs., COLLECT_GCC=gcc, COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/14/lto-wrapper, Target: x86_64-pc-linux-gnu, Configured with: /var/tmp/portage/sys-devel/gcc-14.1.1_p20240615/work/gcc-14-20240615/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/14 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14 --disable-silent-rules --disable-dependency-tracking --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/14/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 14.1.1_p20240615 p2' --with-gcc-major-version-only --enable-libstdcxx-time --enable-lto --disable-libstdcxx-pch --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --with-multilib-list=m64 --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libssp --disable-libada --enable-cet --disable-systemtap --disable-valgrind-annotations --disable-vtable-verify --disable-libvtv --with-zstd --without-isl --enable-default-pie --enable-host-pie --enable-host-bind-now --enable-default-ssp --disable-fixincludes --with-build-config='bootstrap-O3 bootstrap-cet', Thread model: posix, Supported LTO compression algorithms: zlib zstd, gcc version 14.1.1 20240615 (Gentoo 14.1.1_p20240615 p2)
[sagemath_environment-10.4.rc1] No stamp file for package 'sagemath_environment' in /mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/var/lib/sage/installed
[sagemath_environment-10.4.rc1] No spkg-legacy-uninstall script; nothing to do
[sagemath_environment-10.4.rc1] [spkg-install] * Creating isolated environment: venv+pip...
[sagemath_environment-10.4.rc1] [spkg-install] * Installing packages in isolated environment:
[sagemath_environment-10.4.rc1] [spkg-install]   - setuptools >= 68.1.1
[sagemath_environment-10.4.rc1] [spkg-install]   - wheel >=0.36.2
[sagemath_environment-10.4.rc1] [spkg-install] > /mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/bin/python3 -m pip
[sagemath_environment-10.4.rc1] [spkg-install]   --python /tmp/build-env-wch4ns_y/bin/python install --use-pep517 --no-warn-
[sagemath_environment-10.4.rc1] [spkg-install]   script-location --no-compile -r /tmp/build-reqs-h7n4fa42.txt
[sagemath_environment-10.4.rc1] [spkg-install] < Looking in links: file:///mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-
[sagemath_environment-10.4.rc1] [spkg-install]   python3.12/var/lib/sage/wheels
[sagemath_environment-10.4.rc1] [spkg-install] < ERROR: Could not find a version that satisfies the requirement
[sagemath_environment-10.4.rc1] [spkg-install]   setuptools>=68.1.1 (from versions: none)
[sagemath_environment-10.4.rc1] [spkg-install] < ERROR: No matching distribution found for setuptools>=68.1.1
[sagemath_environment-10.4.rc1] [spkg-install] 
[sagemath_environment-10.4.rc1] [spkg-install] Traceback (most recent call last):
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/__main__.py", line 178, in _handle_build_error
[sagemath_environment-10.4.rc1] [spkg-install]     yield
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/__main__.py", line 429, in main
[sagemath_environment-10.4.rc1] [spkg-install]     built = build_call(
[sagemath_environment-10.4.rc1] [spkg-install]             ^^^^^^^^^^^
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/__main__.py", line 268, in build_package_via_sdist
[sagemath_environment-10.4.rc1] [spkg-install]     sdist = _build(isolation, srcdir, outdir, 'sdist', config_settings, skip_dependency_check, installer)
[sagemath_environment-10.4.rc1] [spkg-install]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/__main__.py", line 170, in _build
[sagemath_environment-10.4.rc1] [spkg-install]     return _build_in_isolated_env(srcdir, outdir, distribution, config_settings, installer)
[sagemath_environment-10.4.rc1] [spkg-install]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/__main__.py", line 135, in _build_in_isolated_env
[sagemath_environment-10.4.rc1] [spkg-install]     env.install(builder.build_system_requires)
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/env.py", line 136, in install
[sagemath_environment-10.4.rc1] [spkg-install]     self._env_backend.install_requirements(requirements)
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/env.py", line 265, in install_requirements
[sagemath_environment-10.4.rc1] [spkg-install]     run_subprocess(cmd)
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/site-packages/build/_ctx.py", line 71, in run_subprocess
[sagemath_environment-10.4.rc1] [spkg-install]     subprocess.run(cmd, capture_output=True, check=True, env=env)
[sagemath_environment-10.4.rc1] [spkg-install]   File "/usr/lib/python3.12/subprocess.py", line 571, in run
[sagemath_environment-10.4.rc1] [spkg-install]     raise CalledProcessError(retcode, process.args,
[sagemath_environment-10.4.rc1] [spkg-install] subprocess.CalledProcessError: Command '['/mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/bin/python3', '-m', 'pip', '--python', '/tmp/build-env-wch4ns_y/bin/python', 'install', '--use-pep517', '--no-warn-script-location', '--no-compile', '-r', '/tmp/build-reqs-h7n4fa42.txt']' returned non-zero exit status 1.
[sagemath_environment-10.4.rc1] [spkg-install] 
[sagemath_environment-10.4.rc1] [spkg-install] ERROR Command '['/mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/bin/python3', '-m', 'pip', '--python', '/tmp/build-env-wch4ns_y/bin/python', 'install', '--use-pep517', '--no-warn-script-location', '--no-compile', '-r', '/tmp/build-reqs-h7n4fa42.txt']' returned non-zero exit status 1.
[sagemath_environment-10.4.rc1] [spkg-install] ************************************************************************************************************************************************************************************************************************************************************
[sagemath_environment-10.4.rc1] [spkg-install] Failure building sdist and wheel
[sagemath_environment-10.4.rc1] [spkg-install] ************************************************************************************************************************************************************************************************************************************************************
[sagemath_environment-10.4.rc1] ************************************************************************
[sagemath_environment-10.4.rc1] Error installing package sagemath_environment-10.4.rc1
[sagemath_environment-10.4.rc1] ************************************************************************
[sagemath_environment-10.4.rc1] Please email sage-devel (http://groups.google.com/group/sage-devel)
[sagemath_environment-10.4.rc1] explaining the problem and including the log files
[sagemath_environment-10.4.rc1]   /mnt/opt/Sage/sage-dev/logs/pkgs/sagemath_environment-10.4.rc1.log
[sagemath_environment-10.4.rc1] and
[sagemath_environment-10.4.rc1]   /mnt/opt/Sage/sage-dev/config.log
[sagemath_environment-10.4.rc1] Describe your computer, operating system, etc.
[sagemath_environment-10.4.rc1] If you want to try to fix the problem yourself, *don't* just cd to
[sagemath_environment-10.4.rc1] /mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/var/tmp/sage/build/sagemath_environment-10.4.rc1 and type 'make' or whatever is appropriate.
[sagemath_environment-10.4.rc1] Instead, the following commands setup all environment variables
[sagemath_environment-10.4.rc1] correctly and load a subshell for you to debug the error:
[sagemath_environment-10.4.rc1]   (cd '/mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/var/tmp/sage/build/sagemath_environment-10.4.rc1' && '/mnt/opt/Sage/sage-dev/sage' --buildsh)
[sagemath_environment-10.4.rc1] When you are done debugging, you can type "exit" to leave the subshell.
[sagemath_environment-10.4.rc1] ************************************************************************
make[4]: *** [Makefile:3606: sagemath_environment-SAGE_VENV-no-deps] Error 1
make[3]: *** [Makefile:3606: /mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/var/lib/sage/installed/sagemath_environment-10.4.rc1] Error 2
make[2]: *** [Makefile:3062: all-build] Error 2
make[2]: Leaving directory '/mnt/opt/Sage/sage-dev/build/make'
***************************************************************
Error building Sage.

So an optional package (sagemath_environment got promoted to optional) fails to build.

sagemath / sage