sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.46k stars 485 forks source link

Upgrade: pari 2.13 #30801

Closed slel closed 3 years ago

slel commented 4 years ago

This is to upgrade to PARI 2.13.x. https://repology.org/project/pari/versions

This new PARI release brings a lot of bug fixes, new functionality, and speedups.

Changes in the PARI library that needs adaptation in SageMath code

 34- bnfissunit is obsolete, use bnfisunit
 35- bnfsunit is mostly obsolete, use bnfunits
 36- bnfisunit and bnfissunit: torsion unit exponent is now a t_INT (used
     to be a t_INTMOD)
 1- Removed member functions .futu and .tufu [deprecated since 2.2], used in simon two descent scripts.
 56- zetamultall: add flag

Last upgrade:

Follow-up ticket: #31754

Upstream: Fixed upstream, but not in a stable release.

CC: @dimpase @orlitzky @mkoeppe @slel @antonio-rojas @kiwifb @videlec @tobihan @collares @isuruf @dkwo @xcaruso @loefflerd @kliem

Component: packages: standard

Keywords: upgrade, pari

Author: Vincent Delecroix, Antonio Rojas, Gonzalo Tornaría

Branch: c78b147

Reviewer: Dima Pasechnik, David Loeffler

Issue created by migration from https://trac.sagemath.org/ticket/30801

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 4e43afd to b53bb87

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b53bb87upgrade cypari2 to 2.1.2
videlec commented 4 years ago
comment:43

Remaining elementary doctest failures

sage -t --random-seed=0 schemes/elliptic_curves/ell_rational_field.py  # 1 doctest failed
sage -t --random-seed=0 schemes/elliptic_curves/ell_number_field.py  # 3 doctests failed
sage -t --random-seed=0 doctest/sources.py  # 1 doctest failed
sage -t --random-seed=0 misc/sagedoc.py  # 4 doctests failed
sage -t --random-seed=0 rings/number_field/number_field.py  # 1 doctest failed
sage -t --random-seed=0 schemes/elliptic_curves/isogeny_small_degree.py  # 1 doctest failed
sage -t --random-seed=0 schemes/elliptic_curves/gal_reps_number_field.py  # 1 doctest failed
sage -t --random-seed=0 modules/torsion_quadratic_module.py  # 3 doctests failed
sage -t --random-seed=0 arith/misc.py  # 2 doctests failed
sage -t --random-seed=0 groups/fqf_orthogonal.py  # 2 doctests failed
sage -t --random-seed=0 quadratic_forms/genera/genus.py  # 2 doctests failed
sage -t --random-seed=0 doctest/forker.py  # 1 doctest failed
sage -t --random-seed=0 rings/polynomial/polynomial_element.pyx  # 1 doctest failed
sage -t --random-seed=0 schemes/elliptic_curves/ell_finite_field.py  # 3 doctests failed
sage -t --random-seed=0 schemes/toric/chow_group.py  # 3 doctests failed
sage -t --random-seed=0 rings/finite_rings/finite_field_constructor.py  # 2 doctests failed
sage -t --random-seed=0 rings/number_field/number_field_ideal.py  # 4 doctests failed
sage -t --random-seed=0 schemes/elliptic_curves/ell_generic.py  # 1 doctest failed
sage -t --random-seed=0 modular/local_comp/smoothchar.py  # 2 doctests failed
sage -t --random-seed=0 rings/finite_rings/residue_field.pyx  # 7 doctests failed
sage -t --random-seed=0 rings/number_field/number_field_rel.py  # 2 doctests failed
sage -t --random-seed=0 modules/fg_pid/fgp_module.py  # 33 doctests failed
sage -t --random-seed=0 rings/number_field/order.py  # 1 doctest failed
sage -t --random-seed=0 rings/finite_rings/integer_mod_ring.py  # 1 doctest failed
sage -t --random-seed=0 schemes/elliptic_curves/period_lattice.py  # 1 doctest failed
sage -t --random-seed=0 modular/modsym/p1list_nf.py  # 1 doctest failed
sage -t --random-seed=0 rings/polynomial/polynomial_quotient_ring.py  # 1 doctest failed
sage -t --random-seed=0 groups/abelian_gps/abelian_group.py  # 1 doctest failed
sage -t --random-seed=0 schemes/toric/points.py  # 9 doctests failed
sage -t --random-seed=0 libs/pari/tests.py  # 2 doctests failed
sage -t --random-seed=0 rings/number_field/number_field_ideal_rel.py  # 1 doctest failed
sage -t --random-seed=0 rings/number_field/S_unit_solver.py  # 8 doctests failed
sage -t --random-seed=0 modular/local_comp/liftings.py  # 1 doctest failed
sage -t --random-seed=0 schemes/toric/homset.py  # 3 doctests failed
sage -t --random-seed=0 all.py  # 1 doctest failed
sage -t --random-seed=0 lfunctions/pari.py  # 1 doctest failed
sage -t --random-seed=0 rings/number_field/unit_group.py  # 10 doctests failed
sage -t --random-seed=0 tests/books/computational-mathematics-with-sagemath/linalg_doctest.py  # 1 doctest failed
sage -t --random-seed=0 rings/number_field/class_group.py  # 2 doctests failed
sage -t --random-seed=0 modules/fg_pid/fgp_morphism.py  # 5 doctests failed
sage -t --random-seed=0 structure/factorization.py  # 4 doctests failed
sage -t --random-seed=0 tests/books/judson-abstract-algebra/galois-sage.py  # 1 doctest failed
sage -t --random-seed=0 modules/free_module_morphism.py  # 3 doctests failed
sage -t --random-seed=0 groups/additive_abelian/additive_abelian_group.py  # 10 doctests failed
sage -t --random-seed=0 modules/fg_pid/fgp_element.py  # 8 doctests failed
sage -t --random-seed=0 libs/pari/__init__.py  # 1 doctest failed
sage -t --random-seed=0 docs/conf.py  # 1 doctest failed
sage -t --random-seed=0 matrix/matrix1.pyx  # 1 doctest failed
sage -t --random-seed=0 rings/number_field/number_field_element.pyx  # 3 doctests failed
sage -t --random-seed=0 rings/integer.pyx  # 1 doctest failed
tobihan commented 4 years ago
comment:44

Using sage 9.2 with the patch from this ticket, when spkg-configure is checking whether the pari 2.13 Debian package is usable, the check

checking whether bnfisunit bug of pari 2.11.3 is fixed...

now returns no. The check passed with Debian's pari 2.11.4.

dimpase commented 4 years ago
comment:45

Replying to @tobihan:

Using sage 9.2 with the patch from this ticket, when spkg-configure is checking whether the pari 2.13 Debian package is usable, the check

checking whether bnfisunit bug of pari 2.11.3 is fixed...

now returns no. The check passed with Debian's pari 2.11.4.

yes, this the question of correct output format, which changed, see #30801 comment:6

I've opened #30906 to fix this (the plan is to allow both formats, old and new)

tobihan commented 4 years ago
comment:46

I see, thanks.

mkoeppe commented 4 years ago
comment:47

build/pkgs/cypari/checksums.ini needs upstream_url. Also, perhaps the spkg should be renamed to cypari2?

tobihan commented 4 years ago
comment:49

Replying to @antonio-rojas:

I'm also experiencing a hang in sage/tests/parigp.py, at

sage: pari('K = bnfinit(y^4-52*y^2+26,1); pol = rnfkummer(bnrinit(K,3,1),Mat(5)); L = rnfinit(K, pol); polredabs(polredbest(L.polabs))')  # long time

Running the same code in gp gives a stack overflow

? pol = rnfkummer(bnrinit(K,3,1),Mat(5))
  ***   at top-level: pol=rnfkummer(bnrinit(K,3,1),Mat(5))
  ***                     ^--------------------------------
  *** rnfkummer: the PARI stack overflows !
  current stack size: 8000000 (7.629 Mbytes)
  [hint] set 'parisizemax' to a nonzero value in your GPRC

After setting parisizemax to 32000000, the rnfinit(K, pol) also seems to hang directly in gp.

tobihan commented 4 years ago
comment:50

Here is the upstream bug report for the timeout, with an answer from Bill Allombert: https://pari.math.u-bordeaux.fr/cgi-bin/bugreport.cgi?bug=2259

videlec commented 3 years ago

Changed dependencies from #30849 to #30849, #31029

videlec commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,11 +1,9 @@
-This is to upgrade to PARI 2.13.0 and cypari to 2.1.2.
+This is to upgrade to PARI 2.13.0

 This new PARI release brings a lot of bug fixes,
 new functionality, and speedups.

 - [pari-2.13 release announcement](http://pari.math.u-bordeaux.fr/archives/pari-announce-20/msg00006.html)
-
-- [cypari2 2.1.2 on PyPI](https://files.pythonhosted.org/packages/22/d9/4374baf5749257362f6d163096f9f5a3fa2a6f231edbe0d61c7bdf281d6e/cypari2-2.1.2.tar.gz)

 Changes in the PARI library that needs adaptation in [SageMath](../wiki/SageMath) code
videlec commented 3 years ago
comment:51

I decided to split cypari2 upgrade which is ready to go #31029.

seblabbe commented 3 years ago
comment:52

I don't know if it can be useful, but archlinux has a patch called sagemath-pari-2.13.patch here: https://aur.archlinux.org/cgit/aur.git/tree/?h=sagemath-git and debian too here: https://sources.debian.org/patches/sagemath/9.2-2/ (links from #31016)

antonio-rojas commented 3 years ago
comment:53

Replying to @seblabbe:

I don't know if it can be useful, but archlinux has a patch called sagemath-pari-2.13.patch here: https://aur.archlinux.org/cgit/aur.git/tree/?h=sagemath-git and debian too here: https://sources.debian.org/patches/sagemath/9.2-2/ (links from #31016)

The Arch one is essentially this branch plus some trivial test output fixes to account for the new Smith normal form output. The main remaining blocker is comment:13 - adapt the free abelian group quotient construction to the Smith normal form changes. Does anybody know where the quotient map is defined?

mkoeppe commented 3 years ago

Changed dependencies from #30849, #31029 to none

mkoeppe commented 3 years ago
comment:54

Needs rebase

mkoeppe commented 3 years ago
comment:55

Any progress here?

antonio-rojas commented 3 years ago

Changed branch from u/vdelecroix/pari-2.13.0-cypari-2.1.2 to u/arojas/pari-2.13.0-cypari-2.1.2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

604dfe4Update pari to 2.13.1
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b53bb87 to 604dfe4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

574f98dFix merge
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 604dfe4 to 574f98d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

3b148faUpdate doctests
b6c0c9aFix doctest failures caused by different choices of generators
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 574f98d to b6c0c9a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

e99a203Fix more tests
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b6c0c9a to e99a203

tornaria commented 3 years ago
comment:61

Can you merge https://github.com/sagemath/sagetrac-mirror/commit/ad702bf59ea7372ce12c37f53a18662cd9baee0b so that system pari-2.13 is accepted?

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f79004dMerge branch 'develop' of git://git.sagemath.org/sage into t/30801/pari-2.13.0-cypari-2.1.2
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e99a203 to f79004d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4d92bcado the bnfisunit check in Pari, not rely on output
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from f79004d to 4d92bca

tornaria commented 3 years ago
comment:64

Thanks. This is what I got with e99a203 using system pari-2.13.1 (voidlinux).

----------------------------------------------------------------------
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/ell_number_field.py  # 3 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/gal_reps_number_field.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/isogeny_small_degree.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/doc/en/thematic_tutorials/sandpile.rst  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/quadratic_forms/genera/genus.py  # 2 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/toric/chow_group.py  # 3 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/ell_generic.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modular/local_comp/smoothchar.py  # 2 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/nf_galois_groups.rst  # 5 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/rings/number_field/number_field_rel.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modular/modsym/p1list_nf.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modules/torsion_quadratic_module.py  # 3 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/groups/fqf_orthogonal.py  # 2 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/period_lattice.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/rings/number_field/number_field_ideal.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/elliptic_curves/ell_finite_field.py  # 3 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modules/fg_pid/fgp_module.py  # 30 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/toric/points.py  # 9 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/groups/abelian_gps/abelian_group.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/libs/pari/tests.py  # 2 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/schemes/toric/homset.py  # 3 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modular/local_comp/liftings.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/structure/factorization.py  # 4 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/tests/books/computational-mathematics-with-sagemath/linalg_doctest.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modules/fg_pid/fgp_morphism.py  # 5 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/tests/books/judson-abstract-algebra/galois-sage.py  # 1 doctest failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/groups/additive_abelian/additive_abelian_group.py  # 10 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/modules/fg_pid/fgp_element.py  # 8 doctests failed
sage -t --warn-long 31.3 --random-seed=0 src/sage/libs/pari/__init__.py  # 1 doctest failed
----------------------------------------------------------------------

New commits:

f79004dMerge branch 'develop' of git://git.sagemath.org/sage into t/30801/pari-2.13.0-cypari-2.1.2
4d92bcado the bnfisunit check in Pari, not rely on output
antonio-rojas commented 3 years ago
comment:65

Yes, this still needs a lot of work. In particular, someone needs to look into comment:13

tornaria commented 3 years ago
comment:66

Regarding comment:13, it seems to me an issue migth be that smith_form() is not "idempotent" and maybe abelian groups expect it to be, as in:

Sage with pari 2.11:

sage: Matrix([[1,0],[0,6]]).smith_form()                                                                                                                                      
(
[1 0]  [1 0]  [1 0]
[0 6], [0 1], [0 1]
)

Sage with pari 2.13:

sage: Matrix([[1,0],[0,6]]).smith_form()                                                                                                                                      
(
[1 0]  [ 1  1]  [ 1  6]
[0 6], [ 0 -1], [ 0 -1]
)

The mismatch is that the smith form in sage is the reverse of the smith form in pari, and the way this is fixed is not "stable". It just happened to work with pari-2.11 by chance.

If this is the case, then a fix might be to reverse the matrix before passing it to pari, etc. In fact, doing that fixes the issue in comment:13, and it also fixes 9 out of 10 test failures in src/sage/groups/additive_abelian/additive_abelian_group.py. It doesn't seem to fix anything in src/sage/modules/fg_pid/fgp_element.py, or anything else, though.

I didn't really look into additive abelian group, so I might be wrong.

The patch I'm trying is this:

--- a/src/sage/matrix/matrix_integer_dense.pyx
+++ b/src/sage/matrix/matrix_integer_dense.pyx
@@ -2444,7 +2444,8 @@ cdef class Matrix_integer_dense(Matrix_dense):

            :meth:`elementary_divisors`
         """
-        v = self.__pari__().matsnf(1).sage()
+        X = self.matrix_space()([self[i,j] for i in xrange(self._nrows-1,-1,-1) for j in xrange(self._ncols-1,-1,-1)])
+        v = X.__pari__().matsnf(1).sage()
         # need to reverse order of rows of U, columns of V, and both of D.
         D = self.matrix_space()([v[2][i,j] for i in xrange(self._nrows-1,-1,-1) for j in xrange(self._ncols-1,-1,-1)])

@@ -2460,13 +2461,13 @@ cdef class Matrix_integer_dense(Matrix_dense):
             # silly special cases for matrices with 0 columns (PARI has a unique empty matrix)
             U = self.matrix_space(ncols = self._nrows)(1)
         else:
-            U = self.matrix_space(ncols = self._nrows)([v[0][i,j] for i in xrange(self._nrows-1,-1,-1) for j in xrange(self._nrows)])
+            U = self.matrix_space(ncols = self._nrows)([v[0][i,j] for i in xrange(self._nrows-1,-1,-1) for j in xrange(self._nrows-1,-1,-1)])

         if self._nrows == 0:
             # silly special cases for matrices with 0 rows (PARI has a unique empty matrix)
             V = self.matrix_space(nrows = self._ncols)(1)
         else:
-            V = self.matrix_space(nrows = self._ncols)([v[1][i,j] for i in xrange(self._ncols) for j in xrange(self._ncols-1,-1,-1)])
+            V = self.matrix_space(nrows = self._ncols)([v[1][i,j] for i in xrange(self._ncols-1,-1,-1) for j in xrange(self._ncols-1,-1,-1)])

         return D, U, V
tornaria commented 3 years ago
comment:67

The patch in the last comment should be in branch u/tornaria/pari-snf, commit f893a25.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 4d92bca to 9c8e76e

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9c8e76etry to make smith_form() 'idempotent'
antonio-rojas commented 3 years ago
comment:69

Replying to @tornaria:

The patch in the last comment should be in branch u/tornaria/pari-snf, commit f893a25.

Thanks, this looks reasonable and fixes the weirdnesses in abelian group quotients. Unfortunately it introduces a bunch of new test failures due the change of transition matrices, but those seem easy to deal with.

antonio-rojas commented 3 years ago

Changed author from Vincent Delecroix to Vincent Delecroix, Antonio Rojas, Gonzalo Tornaría

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9c8e76e to b2eac8c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

b2eac8cUpdate some tests for new Smith form transition matrices
tornaria commented 3 years ago
comment:71

It probably needs more work. In particular I think there may still be bad cases, so let's think a bit more about this.

My current thinking (after glancing at smith_form_gens() in fgp_module.py) that maybe something stronger is required, namely:

OTOH I do wonder if this is really a strong requirement for correctness or just tradition.

I mean: what exactly is wrong with the answer in comment:13? I mean: (0,-1) is indeed a generator of V/W. Granted we were expecting (0,1) but the answer is still correct, isn't it?


New commits:

b2eac8cUpdate some tests for new Smith form transition matrices
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b2eac8c to 52d48d5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

6cc043fUpdate some tests for different ideal and unit group generators
52d48d5Fix some tests in sage.schemes.elliptic_curves
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 52d48d5 to bb3f783

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

1080683Merge branch 'develop' of git://git.sagemath.org/sage into t/30801/pari-2.13.0-cypari-2.1.2
bb3f783Fix a few more tests
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This is to upgrade to PARI 2.13.0
+This is to upgrade to PARI 2.13.x. https://repology.org/project/pari/versions

 This new PARI release brings a lot of bug fixes,
 new functionality, and speedups.
mkoeppe commented 3 years ago
comment:76

Is the plan to drop support for system PARI < 2.13.1? Then spkg-configure.m4 will need changing

dimpase commented 3 years ago
comment:77

I think it's OK to drop older PARIs. Several Linux distros (Debian, Gentoo, arch (I guess)) and Homebrew already carry PARI 2.13.

It seems that only Conda is behind. Isuru, is Conda's PARI update in the works?

dkwo commented 3 years ago
comment:78

Do you know whether the error isirreducible: symbol not found that I get in #31392 is related to this ticket?