search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 480 forks source link

Update Singular to 3-1-3-3 #10903

Closed malb closed 13 years ago

malb commented 13 years ago

Singular spkg upgrade

Singular 3-1-2 fixes a critical bug, cf. #10902. There are more bugfixes in the newest stable release. We should update as soon as possible.

Sage libsingular upgrade

There are various issues where currRing is not correctly set. Some of these are triggered by refcounting the rings and freeing them when they are no longer needed, see the dependency ticket #11339.

Installation instructions

Depends on #11339

CC: @zimmermann6 @simon-king-jena

Component: packages: standard

Keywords: singular SageDays34 sd34

Author: Burcin Erocal, Martin Albrecht, Volker Braun, Simon King

Reviewer: Martin Albrecht, Volker Braun, Simon King

Merged: sage-4.7.2.alpha4

Issue created by migration from https://trac.sagemath.org/ticket/10903

kiwifb commented 13 years ago
comment:2

Last time I tried libsingular from 3.1.2 didn't play well with sage (sage couldn't build). It is documented somewhere on sage-devel.

kiwifb commented 13 years ago
comment:3

http://groups.google.com/group/sage-devel/browse_thread/thread/9544eb87400b5e43/e2424823682d7d53?lnk=gst&q=singular+francois#e2424823682d7d53http://groups.google.com/group/sage-devel/browse_thread/thread/9544eb87400b5e43/e2424823682d7d53?lnk=gst&q=singular+francois#e2424823682d7d53

kiwifb commented 13 years ago
comment:4

sorry for the double pasting the mouse is vintage.

6bfcfeed-779b-4b16-82e6-63808dde0af0 commented 13 years ago
comment:5

Updating singular to at least 3-1-3 (if it's out...) will also help the ARM port, as it's one of the few packages which still give issues with 4.6.2.

kedlaya commented 13 years ago
comment:6

Indeed 3-1-3 is out now. Already 3-1-2 adds a lot of functionality for polynomials over the integers which would be very helpful to have in Sage!

kiwifb commented 13 years ago
comment:7

Trying to compile libsingular with this version and trying to link it to sage is on my TODO list. The singular 3-1-3 executable as called by the pexpect interface breaks a few doctest because normalization has been changed.

kcrisman commented 13 years ago
comment:8

Just FYI, I'm cutting a p10 shortly with a very small fix for Cygwin in it (see #11550), so if someone if making this package, it would be great to base off that.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:9

Replying to @SnarkBoojum:

Updating singular to at least 3-1-3 (if it's out...) will also help the ARM port, as it's one of the few packages which still give issues with 4.6.2.

Cf. #10810.

If someone touches the Singular spkg, please also add a patch for #11645 in case it should then still be necessary. (I've submitted one upstream, see http://www.singular.uni-kl.de:8002/trac/ticket/352#comment:15).

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago

Changed keywords from singular to singular SageDays34

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:10

Replying to @kcrisman:

Just FYI, I'm cutting a p10 shortly with a very small fix for Cygwin in it (see #11550), so if someone if making this package, it would be great to base off that.

FWIW, the Singular spkg from #11550 will probably become a p12, since #11663 meanwhile provides a p11 (which Karl-Dieter's will have to get rebased on again).

I'll then perhaps provide a p13 based on that fixing #11645. :)

Anyway, I guess upgrading Sage's Singular will be attempted during the Sage/Singular Days in Kaiserslautern at the end of September, so we have some time until then...

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago

Changed keywords from singular SageDays34 to singular SageDays34 sd34

malb commented 13 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,6 @@
 Singular 3-1-2 fixes a critical bug, cf. #10902. We should update as soon as possible

 http://www.singular.uni-kl.de/index.php/singular-download.html
+
+* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-4.spkg
+* **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz)
malb commented 13 years ago
comment:12

I haven't run any tests yet and the changes in the SPKG are not committed (because our patches will probably be accepted upstream before 3-1-4 comes out).

malb commented 13 years ago

Author: Burcin Erocal, Martin Albrecht

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:14

Replying to @malb:

I haven't run any tests yet and the changes in the SPKG are not committed (because our patches will probably be accepted upstream before 3-1-4 comes out).

The spkg name should contain the svn version you are using.

Regarding the patch to module_list.py:

Please don't randomly add all libraries any extension module using Singular might also use to singular_libs.

malb commented 13 years ago
comment:15

doctests hang here:

Trying:
    R = PolynomialRing(ZZ, Integer(5), order='lex', names=('x', 'y', 'a', 'b', 'u',)); (x, y, a, b, u,) = R._first_ngens(5)###line 4461:_sage_    >>> R.<x,y,a,b,u>=PolynomialRing(ZZ, 5, order='lex')

they also crash in other places. Thus, more fun ahead.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:16

Replying to @malb:

doctests hang here:

Trying:
    R = PolynomialRing(ZZ, Integer(5), order='lex', names=('x', 'y', 'a', 'b', 'u',)); (x, y, a, b, u,) = R._first_ngens(5)###line 4461:_sage_    >>> R.<x,y,a,b,u>=PolynomialRing(ZZ, 5, order='lex')

they also crash in other places. Thus, more fun ahead.

How about starting with 3-1-3 and some important upstream patches?

malb commented 13 years ago
comment:17

I'm in Kaiserslautern right now, so I have direct access to the Singular developers. Hence, actually fixing the issues seems like the order of the day.

malb commented 13 years ago

Dependencies: #11339

malb commented 13 years ago
comment:19

With

$ hg qap
trac_11339_refcount_singular_rings.patch
trac_11339_refcount_singular_polynomials.patch
10903_singular-3-1-4.patch

I currently get these doctest failures.

sage -t  "devel/sage-main/sage/rings/polynomial/polynomial_element.pyx"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_ideal.py"
sage -t  "devel/sage-main/sage/rings/polynomial/toy_d_basis.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_sequence.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_element.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_libsingular.pyx"
malb commented 13 years ago
comment:20

With the updated SPKG & patch I get:

Numerical noise, nothing serious

crash

crash

Factorisation over number fields goes horribly wrong

malb commented 13 years ago

Description changed:

--- 
+++ 
@@ -2,5 +2,5 @@

 http://www.singular.uni-kl.de/index.php/singular-download.html

-* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-4.spkg
+* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3.svn-algnumbers.spkg
 * **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz)
malb commented 13 years ago
comment:22

Hans created a new branch of Singular, where the algebraic extensions are reverted to the old implementation. Using this branch, available in SPKG form at:

http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3.svn-algnumbers.spkg

the crashes seem to go away (at least for everything in sage/rings).

malb commented 13 years ago
comment:23

Output of make ptestlong with this patch and #11339 applied:

sage -t  -long -force_lib devel/sage/sage/misc/sageinspect.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/tests/cmdline.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/interfaces/singular.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/crypto/mq/sr.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/elliptic_curves/ell_generic.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/elliptic_curves/ell_curve_isogeny.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/plane_conics/con_field.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/plane_conics/con_finite_field.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/generic/algebraic_scheme.py # 0 doctests failed

i.e., crashes.

malb commented 13 years ago
comment:24

I just updated the spkg such that SAGE_DEBUG="yes" will build Singular with debugging enabled (internal consistency checks, asserts etc.)

malb commented 13 years ago
comment:25

Alright,

on my computer with 

trac_11339_refcount_singular_rings.patch
trac_11339_refcount_singular_polynomials.patch
10903_singular-3-1-4.patch

I'm down to these two:

$ sage -t -long -force_lib "devel/sage/sage/schemes/plane_conics/con_finite_field.py"
    sage: assert all([C.defining_polynomial()(Sequence(C.has_rational_point(point = True)[1])) == 0 for C in c[r::5]]) # long time: 1.6 seconds
Exception raised:
...
TypeError: Coordinates [4, 4, 1] do not define a point on Projective Conic Curve over Finite Field of size 5 defined by x*y + x*z + y*z + z^2
sage -t -long -force_lib "devel/sage/sage/schemes/generic/algebraic_scheme.py"
    sage: A.subscheme([x]) + A.subscheme([y^2 - (x^3+1)])
Expected:
    Closed subscheme of Affine Space of dimension 2 over Rational Field defined by:
    -x^4 + x*y^2 - x
Got:
    Closed subscheme of Affine Space of dimension 2 over Rational Field defined by:
      x^4 - x*y^2 + x

The latter doesn't look like a bug but an improvement actually.

malb commented 13 years ago
comment:26

on sage.math

sage -t  -long -force_lib devel/sage/sage/crypto/mq/sr.py

also crashes.

zimmermann6 commented 13 years ago
comment:27

Martin, good, you are making progress!

Paul

malb commented 13 years ago
comment:28

make ptestlong passes on sage.math! However, it's likely that different people will see different crashes/bugs on different machines (I know that Volker has a different one on his machine). The reason is: the garbage collector may randomly change the currRing (because of #11339). The fix is to add rChangeRing() before whichever Singular function is called and crashes. As far as I understand it, there shouldn't be a Python functionc all between rChangeRing() and the function for which it is done, because those could trigger the gc.

vbraun commented 13 years ago

Description changed:

--- 
+++ 
@@ -3,4 +3,4 @@
 http://www.singular.uni-kl.de/index.php/singular-download.html

 * **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3.svn-algnumbers.spkg
-* **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz)
+* **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz), [attachment: trac_10903_singular-fixes.patch](https://github.com/sagemath/sage-prod/files/10652281/trac_10903_singular-fixes.patch.gz)
malb commented 13 years ago
comment:30

SPKG doesn't build on bsd. Any suggestions?

g++   -o libsingular.so \
        iparith.o mpsr_Tok.o claptmpl.o \
        grammar.o scanner.o attrib.o blackbox.o eigenval_ip.o extra.o fehelp.o feOpt.o ipassign.o ipconv.o ipid.o iplib.o ipprint.o ipshell.o newstruct.o lists.o sdb.o fglm.o interpolation.o silink.o ssiLink.o subexpr.o janet.o wrapper.o libparse.o sing_win.o gms.o pcv.o maps_ip.o walk.o walk_ip.o cntrlc.o misc_ip.o calcSVD.o pipeLink.o Minor.o MinorProcessor.o MinorInterface.o bigintm.o pyobject_setup.o bbcone.o bbfan.o denom_list.o minpoly.o  slInit_Static.o mpsr_Put.o mpsr_PutPoly.o mpsr_GetPoly.o mpsr_sl.o mpsr_Get.o mpsr_GetMisc.o mpsr_Error.o ndbm.o sing_dbm.o -lkernel -L../kernel -L../factory -L../libfac -L/Users/malb/sage-4.7.2.alpha3/local/lib -lsingfac -lsingcf -lntl -lreadline -lgmp -lomalloc
Undefined symbols:
  "_main", referenced from:
      __start in crt1.o
ld: symbol(s) not found
malb commented 13 years ago
comment:31

The reason is that 

ifeq ($(SINGUNAME),ix86Mac-darwin) 
...
endif

doesn't apply any more, but x86_64Mac-darwin is now returned by singuname.sh

strogdon commented 13 years ago
comment:32

Has anyone tried building the SPKG on amd64? Here the build works for x86 but on amd64 I get

make install in Singular
make![2]: Entering directory `/storage/sage/sage-4.7.2.alpha2/spkg/build/singular-3-1-3.svn-algnumbers/src/Singular'
svnversion >svnver
/bin/sh: svnversion: not found
make![2]: *** [svnver] Error 127
make![2]: Leaving directory `/storage/sage/sage-4.7.2.alpha2/spkg/build/singular-3-1-3.svn-algnumbers/src/Singular'
make![1]: *** [install] Error 1
make![1]: Leaving directory `/storage/sage/sage-4.7.2.alpha2/spkg/build/singular-3-1-3.svn-algnumbers/src'
make: *** ![/storage/sage/sage-4.7.2.alpha2/local/bin/Singular-3-1-3] Error 2
Unable to build Singular.
strogdon commented 13 years ago
comment:33

Replying to @strogdon:

Has anyone tried building the SPKG on amd64? Here the build works for x86 but on amd64 I get

Let me first install subversion and then try again!

malb commented 13 years ago
comment:34

the svnversion business should be fixed by the new SPKG:

http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3-3.spkg

malb commented 13 years ago

Description changed:

--- 
+++ 
@@ -2,5 +2,5 @@

 http://www.singular.uni-kl.de/index.php/singular-download.html

-* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3.svn-algnumbers.spkg
+* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3-3.spkg
 * **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz), [attachment: trac_10903_singular-fixes.patch](https://github.com/sagemath/sage-prod/files/10652281/trac_10903_singular-fixes.patch.gz)
malb commented 13 years ago
comment:35

Open issues:

PS: I won't be able to work on this over the next few days, so happy hacking :)

malb commented 13 years ago
comment:36

cf. http://groups.google.com/group/libsingular-devel/browse_thread/thread/147d5f445b609209

zimmermann6 commented 13 years ago
comment:37

Martin, now I managed to compile the Singular spkg, but when starting Sage I get:

--> 512     from sage.rings.polynomial.multi_polynomial_libsingular import MPolynomialRing_libsingular
    513     if m.integral_domain.is_IntegralDomain(base_ring):
    514         if n < 1:

ImportError: /localdisk/tmp/install/sage/local/lib/python2.6/site-packages/sage/rings/polynomial/multi_polynomial_libsingular.so: undefined symbol: _Z12pTotaldegreeP8spolyrecP9sip_sring
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

Should I have imported first the two patches?

Paul

malb commented 13 years ago
comment:38

Paul, do you have these patches applied

trac_11339_refcount_singular_rings.patch
trac_11339_refcount_singular_polynomials.patch
10903_singular-3-1-4.patch
trac_10903_singular-fixes.patch

and sage -b succeeded (after you installed the new SPKG)?

zimmermann6 commented 13 years ago
comment:39

no, I only applied the first two patches (from #11339) since the description of that ticket (#10903) says to first install the spkg, then to apply the patches. Should I first apply all patches?

Paul

malb commented 13 years ago
comment:40

You'll have to 

trac_11339_refcount_singular_rings.patch
trac_11339_refcount_singular_polynomials.patch
10903_singular-3-1-4.patch
trac_10903_singular-fixes.patch

Sorry for the confusion.

vbraun commented 13 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,17 @@
-Singular 3-1-2 fixes a critical bug, cf. #10902. We should update as soon as possible
+# Singular spkg upgrade

-http://www.singular.uni-kl.de/index.php/singular-download.html
+Singular 3-1-2 fixes a critical bug, cf. #10902. There are more bugfixes in the newest stable release. We should update as soon as possible.

-* **Install:** http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3-3.spkg
+# Sage libsingular upgrade
+
+There are various issues where `currRing` is not correctly set. Some of these are triggered by refcounting the rings and freeing them when they are no longer needed, see the dependency ticket #11339.
+
+
+# Installation instructions
+
+* **Install:** 
+http://www.stp.dias.ie/~vbraun/Sage/spkg/singular-3-1-3-3.spkg
+
+* Apply the two dependency patches from #11339
+
 * **Apply:** [attachment: 10903_singular-3-1-4.patch](https://github.com/sagemath/sage/files/ticket10903/10903_singular-3-1-4.patch.gz), [attachment: trac_10903_singular-fixes.patch](https://github.com/sagemath/sage-prod/files/10652281/trac_10903_singular-fixes.patch.gz)
vbraun commented 13 years ago

Changed author from Burcin Erocal, Martin Albrecht to Burcin Erocal, Martin Albrecht, Volker Braun

vbraun commented 13 years ago
comment:41

I upgraded the spkg with the OSX compile fixes. I only replaced the src/ directory with the new upstream package, I haven't compiled it on OSX. The spkg still has some not-yet-checked-in changes.

vbraun commented 13 years ago
comment:42

The rules for using/setting currRing when calling libSingular are:

This is particularly important because the garbage collector can run between any two Python instructions, but not within C/Cython!

In order to identify code where we forget to set currRing, I implemented a function poison_currRing() in sage.libs.singular.singular that will set currRing to NULL, triggering a segfault if we forget to set it back to a meaningful value before calling libSingular. Using the Python debugger hook, this function can be called between each Python command (but not within Cython code).

It can be globally enabled by uncommenting the lines

### Debugging for Singular, see trac #10903
from sage.libs.singular.ring import poison_currRing
sys.settrace(poison_currRing)

at the very end of sage/all.py.

Using this code, I identified and fixed various places in the Sage library of the form

  rChangeCurrRing(_ring)
  self.parent().some_python_method()
  libSingular_func()

Note that calling some python method in-between exits Cython and allows Python to run the garbage collector etc! Not only do you exit Cython when you return from your Cython function, but also if you call Python in-between.

I also ran into an issue where code uses iteritems() to iterate over locals(), globals(), or dictionaries returned by gc.get_referrers(). This can potentially raise a RuntimeError if those dictionaries change during the iteration. Hooking into the python debugger loop is a good way of triggering precisely that. This also happens in the PolyBoRi Python interface, this is now fixed upstream at https://sourceforge.net/tracker/?func=detail&aid=3416946&group_id=210499&atid=1013986

With the attached patch, I can run the whole Sage testsuite without errors while continuosly poisoning currRing. If you do not patch PolyBoRi, you get some otherwise harmless errors from it but no segfaults. Note that the patch does not enable the poisoning by default. I am quite sure that the patch fixes all remaining Sage/Singular segfault issues. Please give it a try!

vbraun commented 13 years ago

Updated patch

vbraun commented 13 years ago
comment:43

Attachment: trac_10903_singular-fixes.patch.gz

vbraun commented 13 years ago

Description changed:

--- 
+++ 
@@ -9,8 +9,8 @@

 # Installation instructions

-* **Install:** 
-http://www.stp.dias.ie/~vbraun/Sage/spkg/singular-3-1-3-3.spkg
+* **Install:** http://www.stp.dias.ie/~vbraun/Sage/spkg/singular-3-1-3-3.spkg (md5sum: 2557ec04e765c971c1ed6ebf3121f8ac)
+

 * Apply the two dependency patches from #11339
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:45

attachment: 10903_singular-3-1-4.patch still needs work w.r.t. module_list.py; I don't know whether all changes for Singular version 3-1-4 (svn) equally apply to 3-1-3-3. The name of the patch should change anyway if we upgrade to the stable version, i.e., (currently) 3-1-3-X.

If this spkg is to get into Sage 4.7.2, the linker issue (cf. #11769) also has to be fixed here. (Otherwise I'll provide a Singular 3-1-1-4.p14 spkg fixing just this.)

There are other issues which may already be fixed by the new version, or could be fixed by incorporating upstream patches and / or making changes to the Sage library; see comments above (or search for other Singular-related tickets in case I didn't add references to all of them here... ;-) ).

It would be much less confusing if everybody -- at least those significantly -- touching the spkg (i.e., adding changes, especially after someone else made such) bumped the patch level; since we apply patches to the vanilla upstream sources, there should (at least) be a p0 patch level anyway. (This applies to other spkgs as well by the way. It doesn't make sense to have many different spkgs with the exact same name floating around, even if md5sums are given, which in general isn't bad, independent of the former.)