relocatable wheel version of package sage_conf

mkoeppe commented 3 years ago

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI. Hence, the pip invocation needs to use a URL.

Instructions for testing:

Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin

Install the wheel in the user scheme:

$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

Run the installed sage.

$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

also build a sagemath-standard wheel
build wheels for manylinux using tox -e docker-manylinux....
- wheel is much too big: 2.3GB
- auditwheel complains: RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:; so we should replace our current abuse of package data (install_data) by platlib (install_lib). https://github.com/pypa/packaging-problems/issues/542#issuecomment-912838470: subclass build_ext instead of build_py
disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible -- for example by storing a tar file sage_root.tar (with symlinks) instead of including the sage root as a directory - https://docs.python.org/3/library/tarfile.html#tarfile.open
31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409, #31521 to #29039, #31409, #31521, #31552, #31562

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 41ac551 to 00e3956

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`89f61d2`	`build/pkgs/giac: Update to 1.6.0.47p3`
`00e3956`	`Merge #31562`

mkoeppe commented 3 years ago

comment:36

Replying to @mkoeppe:

libintl is still coming in through giac

Solved now via #31562.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

e41932a src/pkgs/sage_conf-relocatable/tox.ini (macos): Add more SDK paths

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 00e3956 to e41932a

mkoeppe commented 3 years ago

Work Issues: Refactor through #31567

mkoeppe commented 3 years ago

comment:39

Replying to @mkoeppe:

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local)

freetype checks for harfbuzz using pkg-config, accepts configure --without-harfbuzz. https://github.com/freetype/freetype/blob/f9f6adb625c48ef15b5d61a3ac1709a068ea95a3/builds/unix/configure.raw#L504

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409, #31521, #31552, #31562 to #29039, #31409, #31521, #31552, #31562, #31567

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,8 +10,10 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:

-./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8) +$ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash +[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && /usr/local/bin/tox -v -v -v)

 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

 Follow-up step: Do not package up the wheel-building venv.
+

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e41932a to 4ed1bf4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`87e2dba`	`tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}`
`0458f23`	`tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}`
`39629dd`	`Merge #31552`
`9d9c562`	`Merge #31562`
`a5f1fc7`	`Merge #31567`
`3dd4885`	`build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no`
`68c447f`	`tox.ini (local -- bash): Set PS1`
`4ed1bf4`	`src/pkgs/sage_conf-relocatable: Remove settings that are now set in SAGE_ROOT/tox.ini`

mkoeppe commented 3 years ago

Changed work issues from Refactor through #31567 to none

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@

$ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash -[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && /usr/local/bin/tox -v -v -v) +[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,8 +10,8 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:

-$ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash -[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel) +$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash +(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:

-$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash +$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 4ed1bf4 to e4270dd

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`300d054`	`build/pkgs/pillow/spkg-install.in: Rename environment variable extra_build_ext to PILLOW_BUILD_EXT`
`8c8d83d`	`build/pkgs/freetype/spkg-install.in: Pass environment variable FREETYPE_CONFIGURE to configure`
`12c84a3`	`tox.ini (macos-nohomebrew): Prevent /usr/local from leaking in through boost, freetype, pillow`
`7966b66`	`Merge #31409`
`fadd2b9`	`Merge #31567`
`59f41b9`	`src/pkgs/sage_conf-relocatable/setup.py: Resolve SAGE_ROOT`
`b5edde2`	`tox.ini (macos-nohomebrew): Fix typo in PILLOW_...`
`e15b152`	`Merge #31567`
`dffcdbc`	`tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT`
`e4270dd`	`Merge #31567`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`4069297`	`src/pkgs/sage_conf-relocatable/setup.py: Unlink a previous symlink 'sage_root' in the package source tree before setting the new one`
`3779a25`	`tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew`
`587142f`	`Merge #31567`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e4270dd to 587142f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

c5a823c src/pkgs/sage_conf-relocatable/setup.py: Make sure that we do not package a copy of the sagelib source tree

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 587142f to c5a823c

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -13,7 +13,7 @@
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

-This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985 +This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

Follow-up step: Do not package up the wheel-building venv.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -15,5 +15,5 @@

This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

-Follow-up step: Do not package up the wheel-building venv. +Follow-up step: Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/); disable all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -15,5 +15,5 @@

This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

-Follow-up step: Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/); disable all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible +Follow-up steps: Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/); do not ship any any wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's `/var/tmp` surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

@@ -15,5 +15,14 @@

This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

+In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases; +install the wheel using + + +pip install https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl + +Then proceed with the same instructions as https://pypi.org/project/sage-conf/ + + Follow-up steps: Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/); do not ship any any wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -19,7 +19,7 @@
 install the wheel using

-pip install https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl +/usr/bin/python3 -m pip install --user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl

 Then proceed with the same instructions as https://pypi.org/project/sage-conf/

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -24,5 +24,7 @@
 Then proceed with the same instructions as https://pypi.org/project/sage-conf/

-**Follow-up steps:** Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible
+**Follow-up steps:** 
+- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- split out `sage_local` from `sage_conf` and use install-requires in the `sage_conf` wheels with @ links to GH releases to the `sage_local` wheel and the built non-`any` wheels.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -25,6 +25,6 @@

 **Follow-up steps:** 
-- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`) (#29013); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - split out `sage_local` from `sage_conf` and use install-requires in the `sage_conf` wheels with @ links to GH releases to the `sage_local` wheel and the built non-`any` wheels.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -13,7 +13,7 @@
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

-This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985 +This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.

In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases; install the wheel using @@ -26,5 +26,5 @@

Follow-up steps:

Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/) (#29013); do not ship any any wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; -- split out sage_local from sage_conf and use install-requires in the sage_conf wheels with @ links to GH releases to the sage_local wheel and the built non-any wheels. +- #31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

mkoeppe commented 3 years ago

Changed dependencies from #29039, #31409, #31521, #31552, #31562, #31567 to #29039, #31409, #31521, #31552, #31562, #31567, #29013

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash -(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel) +(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

 This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from c5a823c to a3ccb60

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`30926d8`	`.github/workflows/tox*.yml: Replace ubuntu-focal-i386 by manylinux-2_24-i686`
`a06ccf7`	`Merge #31541`
`bf6e5f8`	`build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV`
`832f96a`	`build/make/Makefile.in: Rework using tree_... variables`
`3436ad6`	`build/make/Makefile.in: Define SAGE_VENV earlier`
`8cc0b3c`	`WIP configure --with-sage-venv`
`1cfbe2b`	`build/bin/sage-spkg: Change more SAGE_LOCAL to SAGE_INST_LOCAL`
`2202f84`	`configure.ac: Finish --with-sage-venv handling`
`fa897c6`	`Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`a3ccb60`	`src/pkgs/sage_conf-relocatable/setup.py: Build the wheels in a venv outside of SAGE_LOCAL`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from a3ccb60 to a295b81

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`b604a68`	`build/make/Makefile.in: Also source sage-src-env-config for script packages`
`92f1a1b`	`build/pkgs/pynac/spkg-install.in: Build with PYTHON_FOR_VENV`
`c94af4b`	`Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`9f1f794`	`src/pkgs/sage_conf-relocatable/setup.py: Include plat_name in the directory name for SAGE_ROOT`
`d19c5a4`	`Fixup`
`5382ec7`	`src/pkgs/sage_conf-relocatable/setup.py: Working version`
`a295b81`	`.gitignore: Add more`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`e4e6a1e`	`src/pkgs/sage_conf-relocatable/setup.py: Fixup`
`323dcf1`	`src/pkgs/sage_conf-relocatable/setup.py: Add first install_requires @ GH`
`4049273`	`src/pkgs/sage_conf-relocatable/setup.py: Add install_requires for all built binary wheels`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from a295b81 to 4049273

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,30 +1,33 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's `/var/tmp` surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

-To test:
+So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.
+
+The wheel declares dependencies (`install-requires`) to all Python packages in the Sage distribution that have extension modules.  The dependencies are specific using `@` to URLs on 
+https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.
+
+**Instructions for testing:**

-./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v) - -Specifically for building a wheel on macOS that uses XCode python 3.8: +$ /usr/bin/python3 -m pip install -U --user sage-conf + + + + +Procedure for building the sage_conf wheel and compatible wheels: + + +For building a wheel on macOS that uses XCode python 3.8:

 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

-This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.

-In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases; -install the wheel using

- -/usr/bin/python3 -m pip install --user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl - -Then proceed with the same instructions as https://pypi.org/project/sage-conf/

Follow-up steps: -- Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/) (#29013); do not ship any any wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; +- do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible;

31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 **Instructions for testing:**

-$ /usr/bin/python3 -m pip install -U --user sage-conf +$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf sage-conf

mkoeppe commented 3 years ago

comment:69

Unfortunately it turns out that PyPI does not allow @ references in install requires (see https://github.com/pypa/pip/issues/6301)

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Can't have direct dependency: 'Cython @ https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/Cython-0.29.21-cp38-cp38-macosx_10_15_x86_64.whl'

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 **Instructions for testing:**

-$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf sage-conf +$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -2,10 +2,11 @@

 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

-So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.
+So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.  

 The wheel declares dependencies (`install-requires`) to all Python packages in the Sage distribution that have extension modules.  The dependencies are specific using `@` to URLs on 
-https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.
+https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.  Unfortunately, by PyPI policy, such `@` references are not allowed for packages on PyPI. 
+Hence, the pip invocation needs to use a URL.

 **Instructions for testing:**

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -14,17 +14,19 @@
 $ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

-

Procedure for building the sage_conf wheel and compatible wheels:

-For building a wheel on macOS that uses XCode python 3.8: +For building wheels on macOS that uses XCode python 3.8:

 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

+ +This creates: +- src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl +- src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`916bda8`	`configure.ac: Add configure option --disable-doc`
`5871e7c`	`Have sagemath_standard wheel depend on sage_conf wheel @ GH`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 4049273 to 5871e7c

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -11,7 +11,9 @@
 **Instructions for testing:**

-$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl +$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin +$ /usr/bin/python3 -m pip install -U --user https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl +$ sage


 **Procedure for building the sage_conf wheel and compatible wheels:**
@@ -29,7 +31,6 @@
 - src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

-
 **Follow-up steps:** 
 - do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 5871e7c to 6a3d380

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`7ff26df`	`git grep -l SAGE_SUITESPARSE \| xargs sed -i .bak s/SAGE_SUITESPARSE_LOCALINSTALL/SAGE_SUITESPARSE_PREFIX/g`
`52537d1`	`build/pkgs/cvxopt/spkg-install.in: Actually use the value of SAGE_SUITESPARSE_PREFIX`
`ed1a089`	`Merge #31584`
`be5b7bc`	`Merge #31567`
`1fb3a9b`	`build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3`
`24ebfb5`	`src/bin/sage: Only do the check for an incomplete installation when run from the source tree`
`f331ee7`	`src/pkgs/sage_conf-relocatable/setup.py: Pass CONFIGURED_{CC,CXX} to configure if set`
`6a3d380`	`src/pkgs/sage_conf-relocatable/setup.py: Package libexec too`

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -32,6 +32,7 @@

 **Follow-up steps:** 
-- do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- build wheels for `manylinux` using `tox -e docker-manylinux....`
+- disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.

sagemath / sage