search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.4k stars 473 forks source link

relocatable wheel version of package sage_conf #31396

Closed mkoeppe closed 1 year ago

mkoeppe commented 3 years ago

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI. Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

  1. Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:
$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin
  1. Install the wheel in the user scheme:
$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
  1. Run the installed sage.
$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -7,6 +7,8 @@
 The wheel declares dependencies (`install-requires`) to all Python packages in the Sage distribution that have extension modules.  The dependencies are specific using `@` to URLs on 
 https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.  Unfortunately, by PyPI policy, such `@` references are not allowed for packages on PyPI. 
 Hence, the pip invocation needs to use a URL.
+
+(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

 **Instructions for testing:**
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 6a3d380 to cabb706

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

cabb706src/pkgs/sage_conf-relocatable/setup.py: configure --without-system-boost...
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from cabb706 to 2cb3814

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

1942363build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3
9dcded5src/bin/sage: Only do the check for an incomplete installation when run from the source tree
6e0c19esage.misc.package.installed_packages: After SAGE_SPKG_INST, also go through SAGE_VENV_SPKG_INST
12d34eeMerge #29013
5161d65build/pkgs/sage_conf/src/sage_conf.py.in: Move SAGE_ROOT, SAGE_LOCAL to beginning of file; only use substitution @prefix@ once
986ca18build/pkgs/sage_conf/src/sage_conf.py.in: replace subst by replace
6f05cc2sage_setup.setenv: New, use it in setup.py so that build works outside of sage-env
e7ff393Merge #31338
2cb3814src/pkgs/sage_conf-relocatable/sage_conf.py.in: Update from src/pkgs/sage_conf/
mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409, #31521, #31552, #31562, #31567, #29013 to #29039, #31409, #31521, #31552, #31562, #31567, #29013, #31338

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 2cb3814 to a19f514

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

c5dd7baMerge tag '9.3.rc2' into t/29013/public/29013-use-py3-venv
f0fb082sage.misc.package.installed_packages: Refactor through new generator _spkg_inst_dirs
d68e861sage.misc.package.package_manifest: Go through all _spkg_inst_dirs()
54333e5Merge branch 't/29013/public/29013-use-py3-venv' into t/31396/relocatable_wheel_version_of_package_sage_conf
a19f514src/pkgs/sage_conf*/README.rst: Unify the files
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

7484ddabuild/pkgs/sagelib/src/README.rst: Link to sage-conf on pypi
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from a19f514 to 7484dda

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409, #31521, #31552, #31562, #31567, #29013, #31338 to #29039, #31562, #31567, #29013, #31338

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 7484dda to e523da7

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

e523da7tox.ini (local-macos-nohomebrew): Add to Python3.framework/Headers to CPATH
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -27,11 +27,22 @@
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

+or + + +tox -e local-macos-10.15-nohomebrew-python3_xcode -- sage-wheels +

This creates:

+Build for Linux: + + +$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels + +

Follow-up steps:

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31562, #31567, #29013, #31338 to #29039, #31562, #31567, #29013, #31338, #31427

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e523da7 to c080bbc

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

37c9fc0Makefile: New targets pypi-sdists, sage-wheels
3a8406eFixup: Add missing symlinks src/pkgs/sage_conf-{pypi,relocatable}/README.rst
82d9f14build/bin/write-dockerfile.sh: ADD some missing files in SAGE_ROOT
7d10bd3tox.ini (docker), build/bin/write-dockerfile.sh: Handle EXTRA_SAGE_PACKAGES
4a24bc5src/pkgs/sage_conf-relocatable/setup.py: Use configure --enable-build-as-root
c92b335build/pkgs/pynac/spkg-install.in: Explicitly use python3 in SAGE_VENV
32c069a.gitignore: Integrate src/{doc,sage}/.gitignore here so that Docker ignores the listed files too
c080bbcMerge #31427
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

927ce20build/bin/sage-spkg: Also source sage-src-env-config so that SAGE_VENV is available
22c214esrc/pkgs/sage_conf-relocatable/setup.py: Use configure --without-system-zeromq
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from c080bbc to 22c214e

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -46,6 +46,7 @@

 **Follow-up steps:** 
 - build wheels for `manylinux` using `tox -e docker-manylinux....`
+  - auditwheel complains: `RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:`; so we should replace our current abuse of package data (`install_data`) by platlib (`install_lib`)
 - disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 22c214e to 79e2ef4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

5ba62ae.gitignore: Ignore spkg/pkgs/*/{build,dist}
0808863src/pkgs/sage_conf-relocatable/setup.py: Move/copy/delete build trees more carefully
79e2ef4tox.ini (manylinux-python3...): Add EXTRA_SAGE_PACKAGES so that system python is accepted
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -46,6 +46,7 @@

 **Follow-up steps:** 
 - build wheels for `manylinux` using `tox -e docker-manylinux....`
+  - wheel is much too big: 2.3GB
   - auditwheel complains: `RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:`; so we should replace our current abuse of package data (`install_data`) by platlib (`install_lib`)
 - disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 79e2ef4 to cc8297c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9a3aa38build/bin/write-dockerfile.sh: ADD some missing files in SAGE_ROOT
1b03fa9src/pkgs/sage_conf-pypi/setup.py: Change SETENV so that it works in shells with set -e
b3b54b7Merge tag '9.3.rc2' into t/29039/pip_installable_package_sage_bootstrap
ec10bc9src/pkgs/sage_conf-pypi/setup.py: Fix for shell behavior on fedora
278c36csrc/pkgs/sage_conf-pypi/setup.py: Use 'configure --enable-build-as-root' so it works in containers too
cc8297cMerge #29039
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

04da2c6build/pkgs/ipywidgets: Patch out declaring install-requires of nbformat and jupyterlab-widgets
94b8d3cMerge #30913
b54e77aMerge #29039
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from cc8297c to b54e77a

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31562, #31567, #29013, #31338, #31427 to #29039, #31562, #31567, #29013, #31338, #31427, #28890

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b54e77a to 4dcd135

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

3f4a8efTrac #28890: Pass --disable-static to autoconf packages by default.
8a8efeeTrac #28890: Don't install PARI's static library.
cd39c76Trac #28890: Don't use sdh_configure to configure zlib; it does not use
9622d83build/pkgs/zlib/spkg-install.in: Fixup
3d701f8build/pkgs/zlib/spkg-install.in: Handle errors from configure
56178abbuild/bin/sage-dist-helpers (sdh_configure): On Cygwin, do not use --disable-static (until #30814 is done)
4dcd135Merge #28890
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

c8ccab6build/bin/sage-dist-helpers: Fixup: = instead of ==
6f7819eMerge #28890
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 4dcd135 to 6f7819e

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 6f7819e to 9ab6db9

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

df2f8d1Merge tag '9.3' into t/29039/pip_installable_package_sage_bootstrap
697e3c1Merge #29039
6ea676dMerge tag '9.3' into t/31338/sage_setup__use_paths_within_sage_local_when_provided_via_sage_conf
9ab6db9Merge #31338
mkoeppe commented 3 years ago

Changed dependencies from #29039, #31562, #31567, #29013, #31338, #31427, #28890 to #29039, #31338, #28890

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31338, #28890 to #29039, #31338, #29617

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9ab6db9 to ce51048

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

133aab7Trac #29617: move MAXIMA_FAS to sage_conf.
9b595c0Trac #29617: modify Kenzo interface to support system ECL.
b4ae35eTrac #29617: treat empty MAXIMA_FAS and KENZO_FAS as None.
2d40f36Trac #29617: upgrade kenzo to v1.1.10.
2078fa6Merge tag '9.4.beta0' into t/31593/configure__paths_within__sage_local___prefix__for_sage_conf
5cbe423Merge #31593
be48dd8Merge #29617
b4b3ec8Merge tag '9.4.beta0' into t/29039/pip_installable_package_sage_bootstrap
e1d2df0Merge tag '9.4.beta1' into t/29039/pip_installable_package_sage_bootstrap
ce51048Merge #29039
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

7e8a7a8Makefile (sage-wheels): Use SETUPTOOLS_USE_DISTUTILS=local
801e792src/pkgs/sage_conf-relocatable/setup.py: Use wheel tags in build directory name too
52d669dsrc/pkgs/sage_conf-relocatable/setup.py: Do not use removed option --without-system-boost
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from ce51048 to 52d669d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 52d669d to 55122f3

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

55122f3build/pkgs/pynac/spkg-install.in: Work around Trac #31920
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 55122f3 to ff58be3

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ff58be3Makefile (sage-wheels): Fix wheel location in final message
mkoeppe commented 3 years ago
comment:105

Installation with /usr/bin/python3 -m pip install -I --user https://github.com/sagemath/sage-wheels/releases/download/9.4.beta1/sage_conf-9.4b1-cp38-cp38-macosx_11_0_x86_64.whl https://github.com/sagemath/sage-wheels/releases/download/9.4.beta1/sagemath_standard-9.4b1-cp38-cp38-macosx_11_0_x86_64.whl fails with:

Collecting argon2-cffi (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->sagemath-standard==9.4b1)
  Downloading https://files.pythonhosted.org/packages/74/fd/d78e003a79c453e8454197092fce9d1c6099445b7e7da0b04eb4fe1dbab7/argon2-cffi-20.1.0.tar.gz (1.8MB)
[...]
Failed to build argon2-cffi
dimpase commented 3 years ago
comment:106

29039 is listed as a dependency, is it an error?

mkoeppe commented 3 years ago
comment:107

No, that's correct

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

76023cc.gitignore: Ignore spkg/pkgs/*/{build,dist}
8591334src/pkgs/sage_conf-relocatable/setup.py: Move/copy/delete build trees more carefully
adbbdc1tox.ini (manylinux-python3...): Add EXTRA_SAGE_PACKAGES so that system python is accepted
6a6a66aMakefile (sage-wheels): Use SETUPTOOLS_USE_DISTUTILS=local
64ff2e7src/pkgs/sage_conf-relocatable/setup.py: Use wheel tags in build directory name too
40050a2src/pkgs/sage_conf-relocatable/setup.py: Do not use removed option --without-system-boost
a2aa592build/pkgs/pynac/spkg-install.in: Work around Trac #31920
52ecb27Makefile (sage-wheels): Fix wheel location in final message
84637ddMove sprc/pkgs/sage_conf-relocatable to pkgs/sage-conf_relocatable
1c35e0fPut pkgs/sage-conf in the root
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from ff58be3 to 1c35e0f

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31338, #29617 to #29039