search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 480 forks source link

update M4RI to newest upstream release #9475

Closed malb closed 14 years ago

malb commented 14 years ago

The new version improves elimination to some extent, comes with a cleaner API and has an option to suppress SSE instructions.

Note to the release managers

Apply only m4ri_new_version.v2.patch to the Sage library when merging the new M4RI spkg.

Upstream: Not yet reported upstream; Will do shortly.

CC: @sagetrac-mariah

Component: packages: standard

Keywords: M4RI, spkg-check

Author: Martin Albrecht, Leif Leonhardy

Reviewer: Leif Leonhardy, David Kirkby, Mariah Lenox

Merged: sage-4.5.3.alpha1

Issue created by migration from https://trac.sagemath.org/ticket/9475

malb commented 14 years ago
comment:1

I've uploaded an SPKG to

http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20100701.spkg

This SPKG also takes care of #9381 (SAGE_FAT_BINARY not being respected) and the M4RI part of #9281 (spkg-check)

malb commented 14 years ago
comment:2

I've run the M4RI self test (not the Sage test suite) on the following machines:

a9bc4e33-7b98-4180-affc-8dfcef89e22b commented 14 years ago
comment:3

  1. SPKG.txt under "Releases" says latest release is libm4ri-20100107, do you mean libm4ri-20100701?

  2. Since spkg-check exists, in spkg-install the commented out lines:

# $MAKE check
# if [ $? -ne 0 ]; then
#     echo "libm4ri testsuite failed, please report upstream!"
#     exit 1
# fi

can be removed.

  1. In spkg-install, if SAGE_FAT_BINARY is yes, then sse2 is disabled. What about sse3 as was the reported problem in #9381

  2. This version of libm4ri does not appear under the downloads on the m4ri web site, thus no way to tell if the source in the spkg corresponds to the source under the claimed version.

malb commented 14 years ago
comment:5

Replying to @sagetrac-mariah:

  1. SPKG.txt under "Releases" says latest release is libm4ri-20100107, do you mean libm4ri-20100701?

Fixed.

  1. Since spkg-check exists, in spkg-install  the commented out lines: # $MAKE check # if [ $? -ne 0 ]; then # echo "libm4ri testsuite failed, please report upstream!" # exit 1 # fi can be removed.

Fixed.

  1. In spkg-install, if SAGE_FAT_BINARY is yes, then sse2 is disabled. What about sse3 as was the reported problem in #9381

We never use SSE3, but yeah all checks for SSEx instructions are suppressed in that case.

  1. This version of libm4ri does not appear under the downloads on the m4ri web site, thus no way to tell if the source in the spkg corresponds to the source under the claimed version.

This is because I am upstream and wanted to wait for portability issues before putting the release on the website. I'm putting it online now, if that makes your life easier.

malb commented 14 years ago
comment:6

See 

http://m4ri.sagemath.org

and

http://bitbucket.org/malb/m4ri/wiki/M4RI-20100701 

for the new upstream release.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:7

The changes to SPKG.txt, spkg-install and spkg-check are not checked in.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:8

SPKG.txt:

spkg-install:

spkg-check:

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:9

I'd suggest adding the (standard) section Special Update/Build Instructions to SPKG.txt, with the following:

I've built a p1 spkg with the above mentioned files removed: 392KB vs. 1.2MB

If there are no dependencies on Sage packages (which is the case here), we don't need to add $SAGE_LOCAL/include to the preprocessor search path in spkg-install (CFLAGS and CPPFLAGS).

Perhaps touch src/configure, because it has the same time stamp as configure.*.

Upstream notes:

Tests in progress,

Leif

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:10

I've only installed the new package (dated July 13th) and applied the patch, i.e. did not build Sage from scratch, and haven't (yet) run the testsuite.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:11

Shame on me I missed that one:

./spkg-install: line 47: [x: command not found
if ["x$SAGE_FAT_BINARY" = "xyes"]; then
    SSE2_SUPPORT="--disable-sse2"
else
    SSE2_SUPPORT=""
fi

should be

if [ "x$SAGE_FAT_BINARY" = "xyes" ]; then
    ...

Note that [ is actually a command (namely an alias for or link to test, depending on the shell), and ] is its last parameter.

On both of the above systems, the testsuite passed without errors.

(I reinstalled the package(s) with SAGE_CHECK="yes".)

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:12

Just let me know if I should provide (a) reviewer patch(es) to the spkg (after a commit to the changes not yet checked in); I cannot upload an updated spkg to sage.math though.

Leif

malb commented 14 years ago
comment:13

Replying to @nexttime:

  • Remove upstream Mercurial repository (directory src/.hg, file src/.hgtags)
  • Remove directory src/autom4te.cache, file src/m4ri.vcproj I've built a p1 spkg with the above mentioned files removed: 392KB vs. 1.2MB

It's strange that there was an autom4te.cache, since I rm it in my release script. I'm okay with these changes.

malb commented 14 years ago
comment:14

Leif,

I'd appreciate if you could update SPKG according to your suggestions, I'm okay with them all. If you upload the SPKG somewhere or send it my way I can upload it to sage.math.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:15

Replying to @malb:

I'd appreciate if you could update SPKG according to your suggestions, I'm okay with them all. If you upload the SPKG somewhere or send it my way I can upload it to sage.math. 

Ok, I'll create a cumulative spkg patch and a "stripped-down" p1 package in a few hours and then mail you both.

(I assume your .ac.uk e-mail address in SPKG.txt is still appropriate as upstream contact.)

Currently running stress-test builds of 4.5.rc1... ;-)

Leif

malb commented 14 years ago
comment:16

Yes, that's still current, however martinralbrecht@ googleblablabla might be current for a longer time.

Thanks!

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago

Attachment: trac_9475-libm4ri-20100701.p1-based_on_p0.patch.gz

You have to commit Martin's changes ("p0") first to apply this.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:17

Patch for p1 Sage package is up. Remember to commit Martin's changes (of July 13th) first before you apply this patch. (This patch does not remove the unnecessary files in src/ since they are not under our version control; they are of course deleted from the p1 spkg.)

(Link to) New libm4ri-20100701.p1.spkg is on the way.

Tested with both 4.5.rc0 and rc1 (Ubuntu 9.04 x86_64).

Sorry for the delay.

-Leif

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago

Description changed:

--- 
+++ 
@@ -1 +1,2 @@
-The new version improves elimination to some extend, comes with a cleaner API and has an option to suppress SSE instructions
+The new version improves elimination to some extent, comes with a cleaner API and has an option to suppress SSE instructions.
+
malb commented 14 years ago

Reviewer: Leif Leonhardy, David Kirkby

malb commented 14 years ago
comment:18

I uploaded Leif's SPKG to 

 http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20100701.p1.spkg

and I give it a positive review.

malb commented 14 years ago

Changed author from Martin Albrecht to Martin Albrecht, Leif Leonhardy

malb commented 14 years ago

Upstream: Not yet reported upstream; Will do shortly.

malb commented 14 years ago

Changed reviewer from Leif Leonhardy, David Kirkby to Leif Leonhardy, David Kirkby, Mariah Lenox

89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:20

With a fresh build, sage doesn't start:

Traceback (most recent call last):
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/bin/sage-eval",
line 4, in <module>
   from sage.all import *
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/all.py",
line 73, in <module>
   from sage.matrix.all     import *
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/all.py",
line 1, in <module>
   from matrix_space import MatrixSpace, is_MatrixSpace
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/matrix_space.py",
line 40, in <module>
   import matrix_mod2_dense
ImportError: /mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/matrix_mod2_dense.so:
undefined symbol: mzd_lqup
Sage failed to startup.
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:21

Replying to @rlmill:

With a fresh build, sage doesn't start:

Traceback (most recent call last):
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/bin/sage-eval",line 4, in <module>
   from sage.all import *
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/all.py", line 73, in <module>
   from sage.matrix.all     import *
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/all.py", line 1, in <module>
   from matrix_space import MatrixSpace, is_MatrixSpace
 File "/mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/matrix_space.py", line 40, in <module>
   import matrix_mod2_dense
ImportError: /mnt/usb1/scratch/wstein/build/sage-4.5.1/local/lib/python2.6/site-packages/sage/matrix/matrix_mod2_dense.so:
undefined symbol: mzd_lqup
Sage failed to startup.

Looks as if you haven't applied the Sage library patch (or didn't do sage -b after that).

89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:22

Replying to @nexttime:

Looks as if you haven't applied the Sage library patch (or didn't do sage -b after that).

I've been pretty bad about this lately... Sorry guys.

89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:23
sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"

------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
This probably occurred because a *compiled* component
of Sage has a bug in it (typically accessing invalid memory)
or is not properly wrapped with _sig_on, _sig_off.
You might want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate (sorry).
------------------------------------------------------------
malb commented 14 years ago
comment:24

I tried to reproduce your segmentation fault, but I can't.

malb@sage:~/scratch_sage/sage-4.4$ ./sage -t -long devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
         [16.8 s]

----------------------------------------------------------------------
All tests passed!
Total time for all tests: 16.8 seconds
malb@sage:~/scratch_sage/sage-4.4$ cd devel/sage
malb@sage:~/scratch_sage/sage-4.4/devel/sage$ hg qap
m4ri_new_version.patch
singular-3-1-1-4.patch
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:25

Replying to @rlmill:

sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"

------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
...

Machine, OS, platform? Parallel test? Whole library or just that single file?

89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:26

On geom.math, with a parallel build, in parallel and serial testing.

89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:27
Trying:
    C = mq.MPolynomialSystem(r2).connected_components(); C###line 76:_sage_    >>> C = mq.MPolynomialSystem(r2).connecte
d_components(); C
Expecting:
    [Polynomial System with 16 Polynomials in 16 Variables,
     Polynomial System with 16 Polynomials in 16 Variables]
ok
Trying:
    C[Integer(0)].groebner_basis()###line 80:_sage_    >>> C[0].groebner_basis()
Expecting:
    [x111*x110 + w113*x110 + w113*x112 + w113*x113 + w113*w111 + w113*w112 + x111 + x113 + w110 + w111 + w112,
... (more output)
BOOM
malb commented 14 years ago
comment:28

I executed these steps

Result:

malb@geom:~/scratch_sage/SIGSEGV/sage-4.5.1$ ./sage -t -long devel/sage/sage/crypto/mq/mpolynomialsystem.py 
sage -t -long "devel/sage/sage/crypto/mq/mpolynomialsystem.py"
         [17.1 s]

----------------------------------------------------------------------
All tests passed!
Total time for all tests: 17.1 seconds
89c6e537-b2e3-45e6-882d-d4957b74ffe5 commented 14 years ago
comment:29

Well then, maybe it was a cosmic ray...

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:30

Do I need to apply

http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20100701.p1.spkg

and attachment: m4ri_new_version.patch? Can someone put the ticket number in the commit string?

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:32

Note to the release managers

Apply only m4ri_new_version.v2.patch to the Sage library when merging the new M4RI spkg; that patch is identical to m4ri_new_version.patch except for the commit message. (I don't have the power to replace Martin's attachment.)

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago

Description changed:

--- 
+++ 
@@ -1,2 +1,7 @@
 The new version improves elimination to some extent, comes with a cleaner API and has an option to suppress SSE instructions.

+---
+
+## Note to the release managers
+
+**Apply only [m4ri_new_version.v2.patch](https://github.com/sagemath/sage-prod/files/10650046/m4ri_new_version.v2.patch.gz) to the Sage library** when merging the [new M4RI spkg](http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20100701.p1.spkg).
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:33

Replying to @qed777:

Can someone put the ticket number in the commit string?

Done, but now we have yet another attachment since I couldn't replace Martin's.

(I would have thought his patch's comment was sufficient to conclude that the patch has to be applied to the Sage library repository... ;-) )

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:34

Note: I haven't merged this ticket into 4.5.3.alpha0, because I noticed some segfaults that appear to stem from the new package and/or patch, when I doctested various trial alpha0s on sage.math. At the moment, it seems best to put out a 4.5.3.alpha0 with passing doctests and base on this any necessary efforts to merge the new M4RI into alpha1.

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:35

Of course, all long tests now pass (well, there are no reproducible failures) on sage.math with 4.5.3.alpha0 + #9475, so it seems no new efforts are necessary. I'm checking bsd, redhawk, and t2 now.

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:36

Replying to @qed777:

Of course, all long tests now pass (well, there are no reproducible failures) on sage.math with 4.5.3.alpha0 + #9475, so it seems no new efforts are necessary. I'm checking bsd, redhawk, and t2 now.

The long doctests also pass on bsd, redhawk, and t2.

malb commented 14 years ago
comment:37

There definitely is a bug in this new M4RI, I do get SIGSEGVs on my new laptop. I'll investigate.

malb commented 14 years ago
comment:38
        while(it!=end){
            Exponent e=*it; 
                from_term_map_type::const_iterator from_it=eliminated2row_number.find(e);
                assert(terms_as_exp_step1[row_start[from_it->second]]==e);
                assert(from_it!=eliminated2row_number.end());
 ===>               int index=from_it->second;//...translate e->line number;
                mzd_write_bit(mat_step2_factor,i,index,1);
            it++;
        }

This is where pbori.pyx crashes for me. I installed a new GCC today, so maybe that's to blame?

malb commented 14 years ago
comment:39

I think I got it: TheTrac macro PolyBoRi SPKG in 4.5.2 ships its own M4RI (and Boost) which conflicts with this new M4RI SPKG, thus since Leif removed this redundant copy of M4RI in the PolyBoRi, it works in 4.5.3.alpha0 but not before.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:40

Replying to @malb:

I think I got it: The PolyBoRi SPKG in 4.5.2 ships its own M4RI (and Boost) which conflicts with this new M4RI SPKG, thus since Leif removed this redundant copy of M4RI in the PolyBoRi, it works in 4.5.3.alpha0 but not before.

LOL! (Sometimes little clean-ups make more sense than expected...)

malb commented 14 years ago
comment:41

It seems this ticket is incompatible with#9717. On my laptop I always get SIGSEGVs in pbori.pyx

malb commented 14 years ago
comment:42

I tracked down the issue. The cause is some assumptions inTrac macro PolyBoRi about M4RI which are not met anymore. This ticket can go in I say

malb commented 14 years ago
comment:43

Actually, this ticket can only go in if a fix forTrac macro PolyBoRi is also accepted, cf. #9717

jasongrout commented 14 years ago
comment:44

Minor thing: the documentation for rank still says:

On average 'lqup' should be faster than 'm4ri' and hence it is the default choice.

malb commented 14 years ago
comment:45

9717 has an updated, fixedTrac macro PolyBoRi SPKG

malb commented 14 years ago
comment:46

The updated patch only replaces the mention of LQUP with PLS