pkgs/sagemath-*/tox.ini: Create environment that uses Sage's python as the basepython, fix sagemath-standard deps

[mkoeppe]

Newer versions of tox seem to provision python 3.10 when the environment python is used, but then our wheels don't necessarily fit.

For this reason, the examples shown in the developer's manual hardcoded the Python version (tox -e py39).

We now define tox environments sagepython in tox.ini, which takes Python as SAGE_VENV/bin/python3 and change the manual to use this environment in examples, such as:

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'

(to be run after make build)

Depends on #33936

CC: @dimpase @kwankyu

Component: refactoring

Author: Matthias Koeppe

Branch: ca9d6ce

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32716

[mkoeppe]

Branch: u/mkoeppe/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: 9db6b2f

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,2 +1,6 @@
-Newer versions of `tox` seem to provision python 3.10 when the environment `python` is used, but then our wheels don't fit.
+Newer versions of `tox` seem to provision python 3.10 when the environment `python` is used, but then our wheels don't necessarily fit.

+For this reason, the examples shown in the developer's manual hardcoded the Python version (`tox -e py39`). 
+
+We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples.
+

[mkoeppe]

New commits:

5ce30b8	pkgs/sage-setup, sagemath-{objects,categories,standard}: Add tox environment sagepython
9db6b2f	src/doc/en/developer/packaging_sage_library.rst: Use sagepython instead of hardcoded py39

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,5 +2,9 @@

 For this reason, the examples shown in the developer's manual hardcoded the Python version (`tox -e py39`). 

-We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples.
+We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples, such as:

+```
+./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'
+```
+

[dimpase]

comment:8

command in the ticket description fails with

WARNING: Url 'file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/../../../../venv/var/lib/sage/wheels' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement sage-conf==9.7.beta1
ERROR: No matching distribution found for sage-conf==9.7.beta1
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -m pip install -rrequirements.txt', 1)
sagepython-sagewheels-nopypi finish: getenv /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi after 0.69 seconds
...

[mkoeppe]

comment:9

Did you run make build as the manual explains?

[mkoeppe]

Description changed:

--- 
+++ 
@@ -7,4 +7,4 @@

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'

-
+(to be run after `make build`)

[dimpase]

comment:11

besides make build, I also needed to apply

diff --git a/build/pkgs/sage_conf/install-requires.txt b/build/pkgs/sage_conf/install-requires.txt
index 83ad7e641fa..3d4ff51d565 100644
--- a/build/pkgs/sage_conf/install-requires.txt
+++ b/build/pkgs/sage_conf/install-requires.txt
@@ -1 +1 @@
-sage-conf ~= 9.5.b6
+sage-conf
diff --git a/build/pkgs/sage_setup/install-requires.txt b/build/pkgs/sage_setup/install-requires.txt
index 01a931f5eb5..9d4666c07f4 100644
--- a/build/pkgs/sage_setup/install-requires.txt
+++ b/build/pkgs/sage_setup/install-requires.txt
@@ -1 +1 @@
-sage-setup ~= 9.5.b6
+sage-setup

that 9.5.b6 seemed to come from an obsolete workaround anyway.

[dimpase]

comment:12

And after these changes, it errors out (much later) with

ModuleNotFoundError: No module named 'primecountpy'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)

[dimpase]

comment:13

the latter is probably a config error, as the module is there and well:

sage: import primecountpy
sage: primecountpy.primecount.prime_pi(100)
25

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

b030e7a

Merge tag '9.7.beta2' into t/32716/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9db6b2f to b030e7a

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c9ca4e4

pkgs/sagemath-standard/tox.ini: Set basepython correctly for all supported sagepython-* environments

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b030e7a to c9ca4e4

[mkoeppe]

comment:16

Thanks for testing. Indeed it didn't work with sagemath-standard because I made a mistake in the tox configuration. Fixed now

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

01bd09c

pkgs/sagemath-{environment,repl}/tox.ini: Add sagepython toxenv

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from c9ca4e4 to 01bd09c

[dimpase]

comment:19

I'm not a big yaml/tompl expert, but why

basepython = {env:SAGE_VENV}/bin/python3

is repeated 7 times? Can't it be set once and for all?

[mkoeppe]

comment:20

The basepython variable has useful defaulting behavior: Environment names that include a pyNM factor configure basepython = pythonN.M etc., see https://tox.wiki/en/latest/config.html#tox-environments

I want to override it only for environments with the sagepython factor. Unfortunately I haven't found a more elegant solution than repeating it for the listed environments.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3839cee

Merge tag '9.7.beta3' into t/32716/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 01bd09c to 3839cee

[dimpase]

comment:22

on Fedora 34:

sage: import sage_conf
sage: sage_conf.VERSION
'9.7.beta3'

but

...
Skipping link: wrong project name (not sage-conf): file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl
  Local files found: /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels/sage_conf-9.7b3-py3-none-any.whl
  Given no hashes to check 0 links for project 'sage-conf': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement sage-conf~=9.5.b6
  ERROR: No matching distribution found for sage-conf~=9.5.b6
  Exception information:
  Traceback (most recent call last):
    File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion

     crit = self.state.criteria[name]
  KeyError: 'sage-conf'

[dimpase]

comment:23

it seems that sage-conf is not looked up in the right place, i.e. in venv/.

[mkoeppe]

comment:24

Do you happen to use --enable-editable in this build?

[dimpase]

comment:25

Replying to @mkoeppe:

Do you happen to use --enable-editable in this build?

no

[mkoeppe]

comment:26

OK, I guess I have to make changes to build/pkgs/sage_conf/install-requires.txt so it does not reject the wheel

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

fe510ef

build/pkgs/sage_{conf,setup}/install-requires: Use ~= 9.7.b3

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3839cee to fe510ef

[dimpase]

comment:29

for some reason, in this branch ./configure was hanging (!) at testing for giac version. I did the following obvious change, and it passed

--- a/build/pkgs/giac/spkg-configure.m4
+++ b/build/pkgs/giac/spkg-configure.m4
@@ -23,5 +23,6 @@ SAGE_SPKG_CONFIGURE([giac], [
         ], [sage_spkg_install_giac=yes])
        ], [sage_spkg_install_giac=yes])
        m4_popdef([GIAC_MIN_VERSION])
+       m4_popdef([GIAC_MAX_VERSION])
     ])
 ])

[mkoeppe]

comment:30

That's very strange

[mkoeppe]

comment:31

Do you want to push this change to the branch here or open a new ticket for it? Either way is fine with me

[dimpase]

comment:32

now something is not right with primecountpy (which works fine with "normal" sage, e.g. ./sage -tp 16 src/sage/functions/prime_pi.pyx passes just fine)

sagepython-sagewheels-nopypi finish: envreport  after 0.16 seconds
sagepython-sagewheels-nopypi installed: DEPRECATION: --find-links option in pip freeze is deprecated. pip 21.2 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/9069.,-f file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels,alabaster==0.7.12,Babel==2.9.1,backcall==0.2.0,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.4,cvxopt==1.2.7,cycler==0.11.0,cypari2==2.1.2,cysignals==1.11.2,Cython==0.29.30,decorator==5.0.9,docutils==0.17.1,entrypoints==0.3,fonttools==4.28.4,fpylll==0.5.7,gmpy2==2.1.1,idna==3.2,imagesize==1.2.0,importlib-metadata==4.8.2,ipykernel==6.6.0,ipython==7.29.0,ipython-genutils==0.2.0,ipywidgets==7.6.5,jedi==0.18.1,Jinja2==3.1.2,jupyter-client==7.1.0,jupyter-core==4.9.1,kiwisolver==1.3.2,MarkupSafe==2.1.1,matplotlib==3.5.1,matplotlib-inline==0.1.2,memory-allocator==0.1.3,mpmath==1.2.1,nest-asyncio==1.5.4,networkx==2.6.3,numpy==1.21.4,packaging==21.3,parso==0.8.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.0.1,pkgconfig==1.5.5,pplpy==0.8.6,prompt-toolkit==3.0.22,ptyprocess==0.5.1,pycparser==2.21,Pygments==2.10.0,pyparsing==3.0.6,python-dateutil==2.8.2,pytz==2021.3,pyzmq==22.3.0,requests==2.26.0,rpy2==3.3.6,sage-conf==9.7b3,sagemath-standard @ file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/.tmp/package/1/sagemath-standard-9.7b3.zip,scipy==1.7.3,six==1.16.0,snowballstemmer==2.1.0,Sphinx==4.4.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,sympy==1.10.1,tornado==6.1,traitlets==5.1.1,tzlocal==2.1,urllib3==1.26.6,wcwidth==0.2.5,widgetsnbextension==3.5.2,zipp==3.5.0
  removing /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/tmp
sagepython-sagewheels-nopypi start: run-test-pre 
sagepython-sagewheels-nopypi run-test-pre: PYTHONHASHSEED='632737325'
sagepython-sagewheels-nopypi finish: run-test-pre  after 0.00 seconds
sagepython-sagewheels-nopypi start: run-test 
sagepython-sagewheels-nopypi run-test: commands[0] | python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)'
setting PATH=/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/src/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/build/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/bin:/home/scratch2/dimpase/mambaforge/bin:/users/dimpase/perl5/bin:/users/dimpase/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/auto/users/dimpase/bin
[3346474] /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard$ /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/all.py", line 138, in <module>
    from sage.algebras.all   import *
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/all.py", line 21, in <module>
    import sage.algebras.catalog as algebras
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/catalog.py", line 84, in <module>
    from sage.algebras.clifford_algebra import CliffordAlgebra as Clifford
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/clifford_algebra.py", line 35, in <module>
    from sage.quadratic_forms.quadratic_form import QuadraticForm
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form.py", line 192, in <module>
    class QuadraticForm(SageObject):
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form.py", line 323, in QuadraticForm
    from sage.quadratic_forms.quadratic_form__local_field_invariants import \
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form__local_field_invariants.py", line 30, in <module>
    from sage.functions.all import sgn
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/functions/all.py", line 68, in <module>
    from .prime_pi import legendre_phi, partial_sieve_function, prime_pi
  File "sage/functions/prime_pi.pyx", line 37, in init sage.functions.prime_pi (build/cythonized/sage/functions/prime_pi.c:5947)
    from primecountpy.primecount import prime_pi as _prime_pi
ModuleNotFoundError: No module named 'primecountpy'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)
sagepython-sagewheels-nopypi finish: run-test  after 0.34 seconds
sagepython-sagewheels-nopypi start: run-test-post 
sagepython-sagewheels-nopypi finish: run-test-post  after 0.00 seconds
_______________________________________________ summary _______________________________________________
ERROR:   sagepython-sagewheels-nopypi: commands failed
cleanup /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/.tmp/package/1/sagemath-standard-9.7b3.zip

iirc I saw this earlier...

[dimpase]

comment:33

Replying to @mkoeppe:

Do you want to push this change to the branch here or open a new ticket for it? Either way is fine with me

as more changes seem to be needed, perhaps just add it here?

[mkoeppe]

comment:34

ok

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

b5fedff

build/pkgs/giac/spkg-configure.m4: Pop what was pushed

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fe510ef to b5fedff

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b5fedff to 74f1e10

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

74f1e10

src/requirements.txt.m4, src/setup.cfg.m4 (install_requires): Add primecountpy

[mkoeppe]

comment:37

Replying to @dimpase:

now something is not right with primecountpy

I have added it as a runtime dependency. Strange, I thought I did this a long time ago already

[dimpase]

comment:39

now this fails with a similar error for lrcalc:

File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/combinat/sf/schur.py", line 21, in <module>
    import sage.libs.lrcalc.lrcalc as lrcalc
  File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/libs/lrcalc/lrcalc.py", line 193, in <module>
    import lrcalc
ModuleNotFoundError: No module named 'lrcalc'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)
sagepython-sagewheels-nopypi finish: run-test  after 0.51 seconds
sagepython-sagewheels-nopypi start: run-test-post 
sagepython-sagewheels-nopypi finish: run-test-post  after 0.00 seconds
_______________________________________________ summary _______________________________________________
ERROR:   sagepython-sagewheels-nopypi: commands failed

[dimpase]

comment:40

I'm trying the change for lrcalc analogous to the one for primecountpy.

[mkoeppe]

Dependencies: #33936

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 74f1e10 to 1c2f41b

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5ea0035	src/requirements.txt.m4, src/setup.cfg.m4 (install_requires): Add lrcalc_python
7b17c2d	Split sage_docbuild.conf to sage.misc.sagedoc_conf
282270f	Fix docs
c2766bf	Move skip_member() to sage_docbuild
4296851	Move the remnants of skip_member from sagedoc_conf
45f9ff3	Merge branch 'develop'
0a02496	Change assert back to if
1c2f41b	Merge #33936

[mkoeppe]

comment:43

Replying to @dimpase:

I'm trying the change for lrcalc analogous to the one for primecountpy.

Indeed, that was missing too, fixed now.

[dimpase]

comment:46

Replying to @dimpase:

I'm trying the change for lrcalc analogous to the one for primecountpy.

that appeared to suffice (but now the branch got updated)

[dimpase]

comment:47

what I don't like are explicit versions of sage-conf and sage-setup in requirements.txt

Obviously there should be an automated way.

[dimpase]

Reviewer: Dima Pasechnik