Modularization of sagelib: Break out separate packages sagemath-environment, sagemath-repl

mkoeppe commented 4 years ago

The distribution sagemath-environment has no dependencies and contains:

sage.env
sage.features
sage.misc.package
sage.misc.package_dir (from #33033)
sage.misc.temporary_file (dependency on SAGE_TMP/lazy_string removed in #33797)
sage.misc.viewer (The basic distributions sagemath-objects and sagemath-categories do not depend on these!)

It also includes the top-level scripts sage and sage-env, and the helper script sage-venv-config. This will allow users to run sage -sh even if sagelib is not installed.

To test:

$ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-environment && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

The distribution sagemath-repl depends on (install-requires) sagemath-environment and sagemath-objects and contains at least the following:

sage.repl
sage.doctest (which depends on sage.repl)
sage.misc.sage_input
sage.misc.sage_eval

It also includes scripts such as sage-runtests (which implements sage -t), sage-ipython (which is called when sage is invoked with no arguments), etc.

Of course, with nothing else installed, neither sage -t nor sage will work fully because sage.all is not available. But sage -t already has the option --environment for specifying a different top-level module in which to run the doctester; we use this in the test suite of sagemath-objects and sagemath-polyhedra (#32432).

To test:

$ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-repl && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Afterwards:

$ source pkgs/sagemath-repl/.tox/py39/bin/activate
(py39) $ sage                                                                                        
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6.rc3, Release Date: 2022-04-29                 │
│ Using Python 3.9.10. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┃ Warning: sage.all is not available; this is a limited REPL.        ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage:

Note the new warning about sage.all.

We also change all pkgs/*/VERSION.txt files from symlinks to regular files, updated by sage-update-version when a new release is made. This makes it easier to cut post-versions such as 9.6.rc3.post4 or dev versions such as 9.6.rc4.dev1, as was necessary during the development of this ticket. See documentation added in this ticket.

Follow-ups (see Meta-ticket #29705):

We rework sagemath-objects, sagemath-categories as namespace packages (#28925).
Change sage-setup so that it install-requires sagemath-environment; then remove the load-path manipulations in setup.py. (#33812)
We change the distributions sagemath-objects and sagemath-categories (from #29865) so that they no longer includes the above modules but instead declare an extras_require "test" on sagemath-repl. (#33812)
Make sage -help, sage -advance show what to install to make options work. Also show what to install on error. (#33795 comment:7)
Add to sagemath-environment: sage.misc.sage_ostools (need to split out Cython stuff), sage.misc.cython (?)
Add an option to make sage_eval work with a smaller module than sage.all, and use this option in most uses of sage_eval (#33802)

Depends on #33797 Depends on #33016 Depends on #33799 Depends on #33801 Depends on #33803

CC: @kwankyu @tobiasdiez @kliem @orlitzky @dimpase @jhpalmieri @kiwifb

Component: refactoring

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: 3884373

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29941

mkoeppe commented 3 years ago

comment:2

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

mkoeppe commented 3 years ago

Changed keywords from none to sd111

mkoeppe commented 3 years ago

comment:4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,15 @@
-This package would package at least the following:
-- `sage.repl` (and hence depends on on basic rings such as `ZZ`, hence on `sage-categories` (#29865))
+The distribution **sagemath-features** depends at most on **sagemath-objects** and contains:
+- `sage.env` 
+- `sage.features`
+(The basic distributions **sagemath-objects** and **sagemath-categories** do not depend on these!)
+
+The distribution **sagemath-repl** depends on (install-requires) **sagemath-features** contains at least the following:
+- `sage.repl` (and hence depends on basic rings such as `ZZ`, hence on `sage-categories` (#29865))
 - `sage.doctest` (which depends on `sage.repl`)
 - `sage.misc.sage_input`
 - ??

+We change the distributions **sagemath-objects** and **sagemath-categories** (#29865) so that they no longer includes the above modules but instead declare an `extras_require` "test" on **sagemath-repl**.
+
+We also rework **sagemath-objects**, **sagemath-categories** as namespace packages.
+

mkoeppe commented 2 years ago

Dependencies: #29865

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 - `sage.features`
 (The basic distributions **sagemath-objects** and **sagemath-categories** do not depend on these!)

-The distribution **sagemath-repl** depends on (install-requires) **sagemath-features** contains at least the following:
+The distribution **sagemath-repl** depends on (install-requires) **sagemath-features** and contains at least the following:
 - `sage.repl` (and hence depends on basic rings such as `ZZ`, hence on `sage-categories` (#29865))
 - `sage.doctest` (which depends on `sage.repl`)
 - `sage.misc.sage_input`

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,9 +1,13 @@
-The distribution **sagemath-features** depends at most on **sagemath-objects** and contains:
+The distribution **sagemath-environment** depends at most on **sagemath-objects** and contains:
 - `sage.env` 
 - `sage.features`
+- `sage.misc.sage_ostools` (?)
+- `sage.misc.cython` (?)
+- `sage.misc.package`
+- `sage.misc.viewer`
 (The basic distributions **sagemath-objects** and **sagemath-categories** do not depend on these!)

-The distribution **sagemath-repl** depends on (install-requires) **sagemath-features** and contains at least the following:
+The distribution **sagemath-repl** depends on (install-requires) **sagemath-environment** and contains at least the following:
 - `sage.repl` (and hence depends on basic rings such as `ZZ`, hence on `sage-categories` (#29865))
 - `sage.doctest` (which depends on `sage.repl`)
 - `sage.misc.sage_input`

mkoeppe commented 2 years ago

Changed dependencies from #29865 to #29865, #32899

mkoeppe commented 2 years ago

Branch: u/mkoeppe/modularization_of_sagelib__break_out_separate_packages_sagemath_environment__sagemath_repl

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

23ecefc pkgs/sagemath-environment: Edit metadata

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Commit: 23ecefc

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c23ffa0 pkgs/sagemath-environment: Edit metadata

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 23ecefc to c23ffa0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c23ffa0 to d2c3a93

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

d2c3a93 src/doc/en/developer/packaging_sage_library.rst: Explain build/pkgs

mkoeppe commented 2 years ago

Author: Matthias Koeppe, ...

mkoeppe commented 2 years ago

Changed dependencies from #29865, #32899 to #29865, #32899, #32977

mkoeppe commented 2 years ago

Changed dependencies from #29865, #32899, #32977 to #29865, #32899, #32977, #32174

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from d2c3a93 to 7cb01cb

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`12a2507`	`src/sage/features/interfaces.py: Fix indentation in docstring`
`0dd76b6`	`Merge tag '9.5.beta7' into t/32174/doctests__detect__safe__external_features_even_if____optional_external__is_not_used`
`07b2512`	`src/sage/misc/latex.py: Conditionalize some doctests on latex_package_tkz_graph`
`cb13437`	`32174: fix the --show-skipped doctest report`
`391dca1`	`sage.misc.latex.png: Mark doctest # optional - imagemagick`
`6f311b3`	`src/sage/doctest/external.py: Revert latex features to 'external'`
`3bfe1a1`	`32174: adapting the doctest listing external softwares`
`827e6f8`	`Merge #32174`
`16cd026`	`pkgs/sagemath-environment/tox.ini: Test all_features`
`7cb01cb`	`src/sage/features/{csdp,lrs}.py: Move imports from sage.cpython inside method`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a216c2d pkgs/sagemath-environment/tox.ini: Also test import sage.misc.package

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 7cb01cb to a216c2d

mkoeppe commented 2 years ago

Changed dependencies from #29865, #32899, #32977, #32174 to #29865, #32899, #32977, #32174, #32671

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`a9355ef`	`pkgs/sagemath-repl, build/pkgs/sagemath_repl: New`
`d3a6fc3`	`build/pkgs/pip: Update to 21.3.1`
`5b001a2`	`build/bin/sage-dist-helpers: Remove '--use-feature=in-tree-build' (default since pip 21.3)`
`f1f4759`	`build/pkgs/distlib: Update to 0.3.3`
`760ff09`	`Merge #32751`
`cbb9d40`	`Merge #32671`
`7ef907f`	`Makefile (pypi-sdists): Add sagemath_environment, sagemath_repl`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a216c2d to 7ef907f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 7ef907f to bcf0c6b

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`1b4a6bf`	`build/pkgs/sagemath_environment/install-requires.txt: New`
`61b9603`	`src/sage/env.py (SAGE_LIB): Fix up`
`bcf0c6b`	`pkgs/sagemath-repl/setup.cfg.m4: Add ipython`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from bcf0c6b to a987182

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`7d964ea`	`pkgs/sagemath-environment/setup.cfg.m4: Add sage_conf as extras_require`
`a987182`	`src/doc/en/developer/packaging_sage_library.rst: Add sage_conf, sagemath-environment in diagram`

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-The distribution **sagemath-environment** depends at most on **sagemath-objects** and contains:
+The distribution **sagemath-environment** has no dependencies and contains:
 - `sage.env` 
 - `sage.features`
 - `sage.misc.sage_ostools` (?)
@@ -7,7 +7,7 @@
 - `sage.misc.viewer`
 (The basic distributions **sagemath-objects** and **sagemath-categories** do not depend on these!)

-The distribution **sagemath-repl** depends on (install-requires) **sagemath-environment** and contains at least the following:
+The distribution **sagemath-repl** depends on (install-requires) **sagemath-environment** and **sagemath-objects** and contains at least the following:
 - `sage.repl` (and hence depends on basic rings such as `ZZ`, hence on `sage-categories` (#29865))
 - `sage.doctest` (which depends on `sage.repl`)
 - `sage.misc.sage_input`

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -13,6 +13,12 @@
 - `sage.misc.sage_input`
 - ??

+To test:
+
+```
+./bootstrap && make build V=0 && ./sage -sh -c '(cd pkgs/sagemath-repl && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'
+```
+
 We change the distributions **sagemath-objects** and **sagemath-categories** (#29865) so that they no longer includes the above modules but instead declare an `extras_require` "test" on **sagemath-repl**.

 We also rework **sagemath-objects**, **sagemath-categories** as namespace packages.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a987182 to b10b4a5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

b10b4a5 build/pkgs/sagemath_{objects,categories}/install-requires.txt: Add version constraints

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from b10b4a5 to 4e5ad96

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4e5ad96 src/sage/all__sagemath_repl.py: New

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 4e5ad96 to fde1f21

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

fde1f21 src/sage/doctest/forker.py: Mark a doctest # optional - sage.symbolic

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0a505a4 src/sage/repl/rich_output/pretty_print.py: Do not fail if sage.plot, sage.graphs cannot be imported

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from fde1f21 to 0a505a4

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,9 +1,10 @@
 The distribution **sagemath-environment** has no dependencies and contains:
 - `sage.env` 
 - `sage.features`
-- `sage.misc.sage_ostools` (?)
+- `sage.misc.sage_ostools` (need to split out Cython stuff)
 - `sage.misc.cython` (?)
 - `sage.misc.package`
+- `sage.misc.temporary_file` (need to remove dependency on `SAGE_TMP`/`lazy_string`)
 - `sage.misc.viewer`
 (The basic distributions **sagemath-objects** and **sagemath-categories** do not depend on these!)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`97999e1`	`pkgs/sagemath-environment/MANIFEST.in: Add sage/misc/temporary_file.py`
`cd34280`	`src/sage, src/sage/misc: Add all__sagemath_environment.py`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0a505a4 to cd34280

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

26e4fff src/sage/repl/rich_output/pretty_print.py: Mark doctests # optional - sage.graphs, sage.plot, sage.symbolic, sage.combinat

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from cd34280 to 26e4fff

mkoeppe commented 2 years ago

Changed dependencies from #29865, #32899, #32977, #32174, #32671 to #29865, #32899, #32977, #32174, #32671, #31306

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`4f827d0`	`src/doc/en/developer/coding_basics.rst: Update section on SAGE_EXTCODE`
`2807d1f`	`src/setup.cfg.m4: Fix up`
`43ca790`	`src/setup.cfg.m4: Fix syntax`
`81e9c9a`	`Merge #32899`
`af47b5f`	`Merge #31386`
`b2cb173`	`src/sage/repl/rich_output/: Move example files here from src/sage/ext_data, use importlib.resources`
`0f63cce`	`src/sage/repl/display/formatter.py: Replace use of SAGE_EXTCODE in doctest by importlib.resources`
`f344c63`	`Merge #31306`
`a98f92b`	`pkgs/sagemath-repl/setup.cfg.m4: Add package_data`
`a9bdf12`	`pkgs/sagemath-repl/setup.cfg.m4: List subpackages`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 26e4fff to a9bdf12

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

7eb48a0 src/sage/repl/ipython_extension.py: Remove .all import in doctest

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a9bdf12 to 7eb48a0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

7c56d4a src/sage/repl/attach.py: In doctest, replace use of SAGE_TMP by tmp_dir()

sagemath / sage

Modularization of sagelib: Break out separate packages sagemath-environment, sagemath-repl #29941