search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.4k stars 473 forks source link

relocatable wheel version of package sage_conf #31396

Closed mkoeppe closed 1 year ago

mkoeppe commented 3 years ago

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI. Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

  1. Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:
$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin
  1. Install the wheel in the user scheme:
$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
  1. Run the installed sage.
$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

mkoeppe commented 3 years ago

Branch: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Commit: 9573b7c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

952fb9aFix up package_data
9573b7cCloser to working
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -2,3 +2,6 @@

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

+Optional steps:
+- Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...
+
mkoeppe commented 3 years ago

Author: Matthias Koeppe

mkoeppe commented 3 years ago
comment:4

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
    /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
    /usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
    /usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink. Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -2,6 +2,13 @@

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

+To test:
+
+```
+(cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)
+```
+On macOS, this builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+
 Optional steps:
 - Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9573b7c to f785d3c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f785d3csrc/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15
mkoeppe commented 3 years ago
comment:6

Using -isysroot helps but giac configuration needs more work - #31403

mkoeppe commented 3 years ago
comment:7

Replying to @mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

mkoeppe commented 3 years ago

Changed dependencies from #29039 to #29039, #31409

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

6da7ba2Add variable TARGETS
b2d4ab8build/pkgs/{r,rpy2}: Downgrade to optional
8732076src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2
dbdf084src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2
737b21csrc/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2
4403924src/sage/interfaces/r.py: Mark all tests # optional - rpy2
e56a2a9Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from f785d3c to e56a2a9

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a228974src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e56a2a9 to a228974

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from a228974 to c260942

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0cac965build/pkgs/ecl: Update to 21.2.1, remove patches included in new version
6a62484delete all merged upstream patches: MR 210, MR 231
62fac00Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf
c260942src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from c260942 to 4f17a04

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

dc5e225build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS
df17ad4tox.ini: Add configuration factors for specific homebrew python3.x
914ea58build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST
986739bm4/sage_check_python_for_venv.m4: Fix reason
0b3e70dbuild/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it
372adcb.github/workflows/tox.yml: Update xcode versions
fc8b676SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting
5ae8044Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf
c5c73a9src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode
4f17a04src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -5,9 +5,14 @@
 To test:

-(cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v) +./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)

-On macOS, this builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+Specifically for building a wheel on macOS that uses XCode python 3.8:
+
+```
+./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8)
+```
+This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.

 Optional steps:
 - Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

8514352src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 4f17a04 to 8514352

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_LOCAL` relocatable using the method proposed in #31076, using `SAGE_LOCAL=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 8514352 to 1baf390

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

c3e4093Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES
4916415Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default
a35428eMerge #30383
e098133src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS
5dbd712src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built
839c534src/pkgs/sage_conf-pypi/setup.py: Fix typo
100ff51src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src
afa2ffdAdd configure option --disable-sagelib
a4f75b8src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README
1baf390Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

f04c134Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
27c4bb9Merge #31409
66c2987build/pkgs/{sage_sws2rst,sage_docbuild}/install-requires.txt: New
7f8ec6cbuild/pkgs/sagelib/src/tox.ini: Add factor sitepackages
0283da5build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system
f720722build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation
c451b31src/setup.cfg.m4 (install_requires): Add sage_conf
6700223Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_
ac0f563Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap
31bba97Merge #29039
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 1baf390 to 31bba97

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f5f4a15src/pkgs/sage_conf-relocatable/setup.py: Build the wheelhouse, but --disable-notebook --disable-r --disable-sagelib
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 31bba97 to f5f4a15

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

@@ -14,6 +14,4 @@

This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.

-Optional steps: -- Also build the wheelhouse (local/var/lib/sage/wheels), at least for tricky packages...

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -14,4 +14,4 @@

This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.

- +Follow-up step: Do not package up the wheel-building venv.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from f5f4a15 to 87db1d0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

87db1d0src/pkgs/sage_conf-relocatable/setup.py: Use --with-mp=gmp instead of broken --without-system-gmp option
mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409 to #29039, #31409, #31521

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

128e8efRevert "do not allow numpy intrinsics when building fat binary"
b871e09Merge #31521
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 87db1d0 to b871e09

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -12,6 +12,6 @@

./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8)

-This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

 Follow-up step: Do not package up the wheel-building venv.
mkoeppe commented 3 years ago
comment:26

delocate-listdeps reports the following library dependencies:

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/gmp/6.2.1/lib/libgmp.10.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/mpfr/4.1.0/lib/libmpfr.6.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/readline/8.1/lib/libreadline.8.1.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zeromq/4.3.3/lib/libzmq.5.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

8b5b273build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2
f71b013Merge #31552
74a0ae0src/pkgs/sage_conf-relocatable/setup.py: Do not package up $SAGE_LOCAL/var/tmp
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b871e09 to 74a0ae0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 74a0ae0 to 5eaaa87

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

aa8ff50src/pkgs/sage_conf-relocatable/setup.py: Use --without-system-readline
5574aa3build/pkgs/giac/spkg-install.in [macOS]: Pass --with-included-gettext so that it does not link to Homebrew libintl
f71ba56configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
5eaaa87src/pkgs/sage_conf-relocatable/setup.py: Configure libgd without freetype
mkoeppe commented 3 years ago
comment:29
/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib
mkoeppe commented 3 years ago
comment:30

libintl is still coming in through giac

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0ccba0cconfigure.ac (--disable-notebook): Disable argon2_cffi
b7051a9configure.ac (--disable-notebook --disable-sagelib): Disable ipython
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 5eaaa87 to b7051a9

mkoeppe commented 3 years ago
comment:32

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local), libgd (which uses libtiff), and pillow (libtiff and everything else).

We could try not to ship matplotlib at all in the wheelhouse - it is responsible for pulling in freetype and pillow - but it still shares one shared library with some of our packages - libpng.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b7051a9 to 41ac551

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

760b6f0configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
0df51e5configure.ac (--disable-notebook): Disable argon2_cffi
42db661configure.ac (--disable-notebook --disable-sagelib): Disable ipython
c63ccb0src/pkgs/sage_conf-relocatable/tox.ini (macos): Also use -isysroot with g++
1d12382src/pkgs/sage_conf-relocatable/setup.py: Use LIBGD_CONFIGURE='--without-...almost everything...'
1ef46d0build/pkgs/giac: Add another patch in patches/autotools
1b0a9eebuild/pkgs/giac: Update to 1.6.0.47p3
41ac551Merge #31562