sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 481 forks source link

update eclib and improve interface for elliptic curve saturation #31443

Closed JohnCremona closed 3 years ago

JohnCremona commented 3 years ago

I recently made a new version of eclib, which will only affect the saturation of points on elliptic curves over Q. The interface for this (in the saturation() method for elliptic curves over Q) has some problems. Mainly, it does not do a good job of choosing the real precision which mwrank should work at, and also (due to shortcomings in eclib which are now improved) (1) when saturation at one prime fails owing to precision being too low, Sage has no way to do something sensible such as raise an error, or put the call into a loop in which the precision is increased; (2) there is no way to saturate at a single prime, or a range of primes, only at all primes or all primes except 2 or all primes up to some bound. The new eclib version allows for a first and last prime at which saturation should be done. The rationale here is that if you have saturated points on one curve and map them to another using an isogeny, then the image points will automatically be saturated at all primes dividing the isogeny degree.

I don't see a sensible way to deal with this in two parts, so I'll make a new eclib spkg (if that is what they are still called) and make the code changes to saturation at the same time.

Incidentally, Sage has its own implementation of the same underlying algorithm for elliptic curves over number fields, as as well as all the above, there should be a parameter called 'implementation' which could be either 'sage' or 'eclib' (or 'pari' before long).

New eclib source tarball at https://github.com/JohnCremona/eclib/releases/download/20210625/eclib-20210625.tar.bz2

Upstream: Fixed upstream, in a later stable release.

CC: @slel @collares @orlitzky @isuruf @kiwifb @antonio-rojas

Component: packages: standard

Keywords: eclib elliptic curve saturation

Author: John Cremona, Dima Pasechnik

Branch: 789550c

Reviewer: Dima Pasechnik, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/31443

JohnCremona commented 3 years ago
comment:80

Updated tarball to eclib-20210408. This has a bugfix but is otherwise compatible with the previous version. The onlt Sage files changed are the package version and checksums, and one docstring (which used to say that one example took 45 minutes, but now it only takes 40 seconds. In Sage-9.3 it will take many hours)

orlitzky commented 3 years ago
comment:81

The version check in build/pkgs/eclib/spkg-configure.m4 needs to be updated to match the new version string.

I also spotted a typo "casue" in sage/libs/eclib/interface.py.

JohnCremona commented 3 years ago
comment:82

Replying to @orlitzky:

The version check in build/pkgs/eclib/spkg-configure.m4 needs to be updated to match the new version string.

I also spotted a typo "casue" in sage/libs/eclib/interface.py.

Thanks, I will fix both right away.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9b1dedbcorrect typo and update eclib's m4 script
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 8c4c115 to 9b1dedb

JohnCremona commented 3 years ago
comment:84

Updated tarball to 20210415, version + checksums + m4 script. Some bugfixes compared with version 20210408. No more Sage code changes needed.

JohnCremona commented 3 years ago

Description changed:

--- 
+++ 
@@ -4,5 +4,4 @@

 Incidentally, Sage has its own implementation of the same underlying algorithm for elliptic curves over number fields, as as well as all the above, there should be a parameter called 'implementation' which could be either 'sage' or 'eclib' (or 'pari' before long).

-New eclib source tarball at 
-[https://github.com/JohnCremona/eclib/releases/download/20210408/eclib-20210318.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210408/eclib-20210318.tar.bz2)
+New eclib source tarball at [https://github.com/JohnCremona/eclib/releases/download/20210415/eclib-20210415.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210415/eclib-20210415.tar.bz2)
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

53a0305Merge remote-tracking branch 'trac/develop' into 31443-eclib-saturation
3dca4e6#31443: undated pkg info for 20210415
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9b1dedb to 3dca4e6

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0ed01dcMerge branch 'develop' into 31443-eclib-saturation
9443892#31443: updated eclib tarball to 20210503
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 3dca4e6 to 9443892

JohnCremona commented 3 years ago

Description changed:

--- 
+++ 
@@ -4,4 +4,4 @@

 Incidentally, Sage has its own implementation of the same underlying algorithm for elliptic curves over number fields, as as well as all the above, there should be a parameter called 'implementation' which could be either 'sage' or 'eclib' (or 'pari' before long).

-New eclib source tarball at [https://github.com/JohnCremona/eclib/releases/download/20210415/eclib-20210415.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210415/eclib-20210415.tar.bz2)
+New eclib source tarball at [https://github.com/JohnCremona/eclib/releases/download/20210503/eclib-20210503.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210503/eclib-20210503.tar.bz2)
JohnCremona commented 3 years ago
comment:87

I am taking the opportunity of the delay to make a small additional update to the eclib tarball (release 20210503), with a minor bugfix. This version also keep debian happy on all its test platforms. No further Sage code changes outside build/pks/eclib.

dimpase commented 3 years ago
comment:88

Is this tested with #30801 (upgrade to Pari 2.13.1) ?

antonio-rojas commented 3 years ago
comment:89

Replying to @dimpase:

Is this tested with #30801 (upgrade to Pari 2.13.1) ?

I've been shipping both downstream for a while without issues

JohnCremona commented 3 years ago
comment:90

Replying to @antonio-rojas:

Replying to @dimpase:

Is this tested with #30801 (upgrade to Pari 2.13.1) ?

I've been shipping both downstream for a while without issues

I have been using recent pari version for eclib development, though eclib does not use very much of pari and nothing at all new.

dimpase commented 3 years ago
comment:91

OK, fine.

JohnCremona commented 3 years ago
comment:92

Is there a reason why this was not included in 9.4.beta0?

slel commented 3 years ago
comment:93

I'm pretty sure it's simply a matter of backlog as many tickets have had positive review for a while.

vbraun commented 3 years ago
comment:94

PDF docs don't build

[docpdf] ! Undefined control sequence.
[docpdf] l.19194 ...q\) of good reduction such that \(E(\FF
[docpdf]                                                   _q)\) has
[docpdf] ? 
[docpdf] ! Emergency stop.
slel commented 3 years ago
comment:95

Change \FF_q to \GF{q}:

-        modulo primes `q` of good reduction such that `E(\FF_q)` has
+        modulo primes `q` of good reduction such that `E(\GF{q})` has
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9443892 to b50d9c6

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

d193588Merge tag '9.3' of github.com:sagemath/sage into public/packages/eclib/31443
b50d9c6Fix doc building
JohnCremona commented 3 years ago
comment:97

Replying to @slel:

Change \FF_q to \GF{q}:

-        modulo primes `q` of good reduction such that `E(\FF_q)` has
+        modulo primes `q` of good reduction such that `E(\GF{q})` has

Thanks for fixing this.

slel commented 3 years ago
comment:98

Thanks Isuru for sending a commit implementing the suggested doc fix.

Back to needs review I guess. Or should it be rebased on 9.4.beta0?

dimpase commented 3 years ago
comment:99

lgtm

vbraun commented 3 years ago

Changed branch from public/packages/eclib/31443 to b50d9c6

vbraun commented 3 years ago

Changed commit from b50d9c6 to none

vbraun commented 3 years ago
comment:101

On 32-bit:

**********************************************************************
File "src/sage/libs/eclib/interface.py", line 555, in sage.libs.eclib.interface.mwrank_EllipticCurve.saturate
Failed example:
    E.gens()
Expected:
    [[-1001107, -4004428, 1]]
Got:
    Attempt to convert -2004462486322 to long fails!
    [[-1001107, -4004428, 1]]
**********************************************************************
1 item had failures:
   1 of   9 in sage.libs.eclib.interface.mwrank_EllipticCurve.saturate
    [192 tests, 1 failure, 18.73 s]
----------------------------------------------------------------------
sage -t --long --random-seed=0 src/sage/libs/eclib/interface.py  # 1 doctest failed
----------------------------------------------------------------------
vbraun commented 3 years ago

Last 10 new commits:

926670fMerge remote-tracking branch 'jc/31443-eclib-saturation' into public/packages/eclib/31443
8b64fedMerge remote-tracking branch 'trac/public/packages/eclib/31443' into 31443-eclib-saturation
8c4c115Merge branch 'public/packages/eclib/31443' of trac.sagemath.org:sage into 31443-eclib-saturation
9b1dedbcorrect typo and update eclib's m4 script
53a0305Merge remote-tracking branch 'trac/develop' into 31443-eclib-saturation
3dca4e6#31443: undated pkg info for 20210415
0ed01dcMerge branch 'develop' into 31443-eclib-saturation
9443892#31443: updated eclib tarball to 20210503
d193588Merge tag '9.3' of github.com:sagemath/sage into public/packages/eclib/31443
b50d9c6Fix doc building
vbraun commented 3 years ago

Commit: b50d9c6

vbraun commented 3 years ago

Changed branch from b50d9c6 to public/packages/eclib/31443

JohnCremona commented 3 years ago
comment:103

Replying to @vbraun:

On 32-bit:

**********************************************************************
File "src/sage/libs/eclib/interface.py", line 555, in sage.libs.eclib.interface.mwrank_EllipticCurve.saturate
Failed example:
    E.gens()
Expected:
    [[-1001107, -4004428, 1]]
Got:
    Attempt to convert -2004462486322 to long fails!
    [[-1001107, -4004428, 1]]
**********************************************************************
1 item had failures:
   1 of   9 in sage.libs.eclib.interface.mwrank_EllipticCurve.saturate
    [192 tests, 1 failure, 18.73 s]
----------------------------------------------------------------------
sage -t --long --random-seed=0 src/sage/libs/eclib/interface.py  # 1 doctest failed
----------------------------------------------------------------------

Are we still supporting 32-bit? What if an upstream package does not support it? I have no way to test eclib on 32-bit.

dimpase commented 3 years ago
comment:104

one can test on 32-bit using GitHub Actions (this would be very easy to set up for standalone eclib), and we still support 32-bit, see http://www.mirrorservice.org/sites/www.sagemath.org/linux/32bit/index.html

JohnCremona commented 3 years ago
comment:105

Replying to @dimpase:

one can test on 32-bit using GitHub Actions (this would be very easy to set up for standalone eclib), and we still support 32-bit, see http://www.mirrorservice.org/sites/www.sagemath.org/linux/32bit/index.html

Thanks, Dima. If you ever feel like helping me set up GitHub actions for eclib, that would be good. I have been using Travis.

JohnCremona commented 3 years ago
comment:106

There is nothing special about the one example which triggers that error on 32-bit machines, and I will just change it to a smaller one. This version of eclib contains may improvements to the code for saturation so it was tempting to put in a non-trivial example in this doctest but it is not necessary.

dimpase commented 3 years ago
comment:107

how does one trigger it? perhaps put this into spkg-check?

dimpase commented 3 years ago
comment:108

there is also a way to make the test output dependent on the cpu, look up special tags 8n the code, # 32-bit if i recall right

JohnCremona commented 3 years ago
comment:109

Yes I know about having 32-bit tests, when I started Sage development I had a 32-bit machine. But now I have not yet been able to build a 32-bit eclib (it is not enough to add -m32 to the compiler flags, it seems, and it might need a 32-bit version of the dependent libraries too).

The only purpose of this test is to show that saturation may not change anything. I'll patch it to use this smaller example instead:

sage: E = mwrank_EllipticCurve([0, 0, 0, -391, 0])                                                                                                                                                                           
sage: E.gens()                                                                                                                                                                                                               
[[196, 2730, 1]]
sage: E.saturate()                                                                                                                                                                                                           
sage: E.gens()                                                                                                                                                                                                               
[[196, 2730, 1]]
dimpase commented 3 years ago
comment:110

I've just built Sage 9.4.beta3 with this ticket on a 32-bit box, and checked that the following works:

--- a/src/sage/libs/eclib/interface.py
+++ b/src/sage/libs/eclib/interface.py
@@ -553,7 +553,9 @@ class mwrank_EllipticCurve(SageObject):

             sage: E = mwrank_EllipticCurve([0, 0, 0, -1002231243161, 0])
             sage: E.gens()
-            [[-1001107, -4004428, 1]]
+            [[-1001107, -4004428, 1]] # 64-bit
+            ...                       # 32-bit
+            [[-1001107, -4004428, 1]] # 32-bit
             sage: E.saturate()
             sage: E.gens()
             [[-1001107, -4004428, 1]]

I can give you access to the box (well, it's old and slow) if you send me your public ssh key.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

7bed91dallow 32-bit boxes to complain
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b50d9c6 to 7bed91d

JohnCremona commented 3 years ago
comment:114

I am tempted let this stand as is, but the warning line which 32-bit machines output could be fatal in that the code has attempted to convert a multiprecision integer to a long int (which is 64-bit or 32-bit, depending), and fails since 2004462486322 has 41 bits. The conversion function returns 0. So the output cannot be trusted, though here it does not give an incorrect result.

I have tracked down exactly where in eclib code this conversion happens. It is in a new piece of code in this version of eclib. It may occur when there are primes of bad reduction greater than (about) 2^31 (or 2^63). A better solution which I can do here is to use multiprecision integers in one place where I did not, so I will make that change and make a new eclib version.

Thanks anyway, Dima.

JohnCremona commented 3 years ago

New commits:

7bed91dallow 32-bit boxes to complain
JohnCremona commented 3 years ago

Description changed:

--- 
+++ 
@@ -4,4 +4,4 @@

 Incidentally, Sage has its own implementation of the same underlying algorithm for elliptic curves over number fields, as as well as all the above, there should be a parameter called 'implementation' which could be either 'sage' or 'eclib' (or 'pari' before long).

-New eclib source tarball at [https://github.com/JohnCremona/eclib/releases/download/20210503/eclib-20210503.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210503/eclib-20210503.tar.bz2)
+New eclib source tarball at [https://github.com/JohnCremona/eclib/releases/download/20210625/eclib-20210625.tar.bz2](https://github.com/JohnCremona/eclib/releases/download/20210625/eclib-20210625.tar.bz2)
JohnCremona commented 3 years ago
comment:116

I'll revert that cosmetic 32-bit fix since after fixing the underlying issue in eclib (new version 20210625) it will not be needed. New patch up soon.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

2c5b93bMerge branch 'develop' into eclib31443
c1871caMerge branch 'public/packages/eclib/31443' of trac.sagemath.org:sage into eclib31443
789550c31443 updated for eclib version 20210625
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 7bed91d to 789550c

JohnCremona commented 3 years ago
comment:118

This version should pass OK on 32-bit and 64.

dimpase commented 3 years ago
comment:119

both on 32 and 64 bits I get this error:

sage -t --random-seed=0 src/sage/rings/number_field/number_field_ideal.py
**********************************************************************
File "src/sage/rings/number_field/number_field_ideal.py", line 2199, in sage.rings.number_field.number_field_ideal.NumberFieldFractionalIdeal.invertible_residues
Failed example:
    list(K.ideal(8).invertible_residues())[:5]
Expected:
    [1, a - 1, -3*a, -2*a + 3, -a - 1]
Got:
    [1, a + 2, 3*a + 3, -2*a + 3, a]

but this does not seem to be related to eclib. Positive review.