Update Singular spkg to release 3-1-1-4

[malb]

The Singular team accepted most of our patches upstream. They are in the 3-1-0-9 release, which also is a first step to make things easier for library developers.

How to apply the patches to sage-4.5.3.alpha0:

• Install the new Singular spkg: http://sage.math.washington.edu/home/mpatel/trac/8059/singular-3-1-1-4.p0.spkg
• Apply singular-3-1-1-4.2.patch to devel/sage

Upstream: Reported upstream. Little or no feedback.

CC: @sagetrac-PolyBoRi @sagetrac-drkirkby @aghitza @jaapspies @kiwifb @alexanderdreyer

Component: packages: standard

Author: Martin Albrecht

Reviewer: David Kirkby, Simon King, William Stein, John Palmieri

Merged: sage-4.5.3.alpha1

Issue created by migration from https://trac.sagemath.org/ticket/8059

[malb]

comment:1

Here's the new SPKG:

http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-0-9-20100125.spkg

You also need to apply the patch I am going to upload in a second.

• CCing Dave so we can make sure there is no regression w.r.t. Solaris etc.
• CCing PolyBoRi so Michael can take a look at my changes (he knows the code base)
• CCing Alex since he reviewed Singular SPKGs before :)

[malb]

Attachment: singular_3-0-1-9.patch.gz

[malb]

comment:3

Doctests on sage.math:

The only failure is in gen.pyx. I don't see how this can be caused by my changes.

**********************************************************************
File "/mnt/usb1/scratch/malb/sage-4.3.1/devel/sage/sage/libs/pari/gen.pyx", line 2950:
    sage: pari(sqrt(2)).frac()
Exception raised:
    Traceback (most recent call last):
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_77[3]>", line 1, in <module>
        pari(sqrt(Integer(2))).frac()###line 2950:
    sage: pari(sqrt(2)).frac()
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/functions/other.py", line 812, in sqrt
        return x.sqrt(*args, **kwds)
      File "integer.pyx", line 4440, in sage.rings.integer.Integer.sqrt (sage/rings/integer.c:25864)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/functions/other.py", line 755, in _do_sqrt
        z = SR(x) ** one_half
      File "expression.pyx", line 2218, in sage.symbolic.expression.Expression.__pow__ (sage/symbolic/expression.cpp:11255)
      File "integer.pyx", line 1603, in sage.rings.integer.Integer.__pow__ (sage/rings/integer.c:11625)
      File "rational.pyx", line 2082, in sage.rings.rational.Rational.__pow__ (sage/rings/rational.c:15560)
    ImportError: /mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/symbolic/power_helper.so: failed to map segment from shared object: Cannot allocate memory

**********************************************************************
File "/mnt/usb1/scratch/malb/sage-4.3.1/devel/sage/sage/libs/pari/gen.pyx", line 2977:
    sage: pari(sqrt(-2)).imag()
Exception raised:
    Traceback (most recent call last):
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_78[3]>", line 1, in <module>
        pari(sqrt(-Integer(2))).imag()###line 2977:
    sage: pari(sqrt(-2)).imag()
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/functions/other.py", line 812, in sqrt
        return x.sqrt(*args, **kwds)
      File "integer.pyx", line 4424, in sage.rings.integer.Integer.sqrt (sage/rings/integer.c:25709)
      File "/mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/functions/other.py", line 755, in _do_sqrt
        z = SR(x) ** one_half
      File "expression.pyx", line 2218, in sage.symbolic.expression.Expression.__pow__ (sage/symbolic/expression.cpp:11255)
      File "integer.pyx", line 1603, in sage.rings.integer.Integer.__pow__ (sage/rings/integer.c:11625)
      File "rational.pyx", line 2082, in sage.rings.rational.Rational.__pow__ (sage/rings/rational.c:15560)
    ImportError: /mnt/usb1/scratch/malb/sage-4.3.1/local/lib/python/site-packages/sage/symbolic/power_helper.so: failed to map segment from shared object: Cannot allocate memory

[malb]

comment:4

Also, the same doctest passes for me locally, so I will assume for now that this has nothing to do with my patch (famous last words, I know).

[aghitza]

comment:6

I'm testing this on a couple of machines and will report.

I have two questions about the spkg:

1. Should we remove dist/, see #5903 ?
2. In src/, should we remove doc/ and emacs/ ? It's really just a question of making the spkg smaller by about 10%, since AFAIK these won't get used from Sage. We routinely do this with other spkgs.

I'm happy to do this and post a new spkg, but I wanted to see what Martin thought about it.

[malb]

comment:7

Replying to @aghitza:

I have two questions about the spkg:

1. Should we remove dist/, see #5903 ?

Yes, that sounds reasonable.

2. In src/, should we remove doc/ and emacs/ ? It's really just a question of making the spkg smaller by about 10%, since AFAIK these won't get used from Sage. We routinely do this with other spkgs.

I just checked and it still builds with these two directories removed.

Thus, I've updated the SPKG accordingly. Same place, same name.

[1998e663-c3b5-4cf6-92c3-7f1771ca5185]

comment:8

I did not have any time to test it. But most changes introduce passing the ring explicitly to Singular. This avoid current ring problems (which can be quite annoying and hard to debug).

nlGetNumerator seems a new addition in Singular (maybe, a special wish of Martin?).

Cheers, Michael

[malb]

comment:9

Replying to @sagetrac-PolyBoRi:

I did not have any time to test it. But most changes introduce passing the ring explicitly to Singular. This avoid current ring problems (which can be quite annoying and hard to debug).

Yep, that's a great direction things are developing.

nlGetNumerator seems a new addition in Singular (maybe, a special wish of Martin?).

It's just nlGetNom renamed

[1998e663-c3b5-4cf6-92c3-7f1771ca5185]

comment:10

Then, what I can still comment are wild casts like

<napoly*>pNext(<poly*>z)

But they are also okay as napoly is just a typedef for polyrec*. So, from reading Malbs code, it looks good. Still someone needs to test it and check it for Sages formal criteria

[aghitza]

comment:11

Replying to @malb:

I just checked and it still builds with these two directories removed.

Thus, I've updated the SPKG accordingly. Same place, same name.

Unless I am looking at the wrong SPKG, you forgot to take dist/ out of version control:

[ghitza@cartan singular-3-1-0-9-20100125]$ hg status
! dist/debian/changelog
! dist/debian/compat
! dist/debian/control
! dist/debian/control.in
! dist/debian/copyright
! dist/debian/libsingular-dev.install
! dist/debian/libsingular0.install
! dist/debian/patches/change-library-search-path.patch
! dist/debian/patches/sage-bugfixes.patch
! dist/debian/patches/series
! dist/debian/rules
! dist/debian/singular.install

Apart from this, I'm happy.

[malb]

comment:12

Replying to @sagetrac-PolyBoRi:

Then, what I can still comment are wild casts like But they are also okay as napoly is just a typedef for polyrec*.

There used to be a naIter function which is gone in 3-1-0-9 and Singular internally uses pNext() as well. Thus I feel my casting is justified :)

[malb]

comment:13

Unless I am looking at the wrong SPKG, you forgot to take dist/ out of version control:

fixed. Is that a positive review then?

[aghitza]

comment:14

Yes.

[jaapspies]

comment:15

As is the spkg failed to build on Open Solaris x64 with SAGE64=yes.

g++ -c cf_factor.cc -w -fno-implicit-templates -I. -I. -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include -DHAVE_CONFIG_H -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include  -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include -O3 -g -fPIC -o cf_factor.o
In file included from /export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include/NTL/vec_ZZ.h:5,
                 from /export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include/NTL/ZZX.h:5,
                 from /export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include/NTL/ZZXFactoring.h:5,
                 from NTLconvert.h:23,
                 from cf_factor.cc:33:
/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include/NTL/ZZ.h: In function ‘long int NTL::MulModPrecon(long int, long int, long int, long unsigned int)’:
/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include/NTL/ZZ.h:1795: error: ‘MulHiUL’ was not declared in this scope
make[2]: *** [cf_factor.o] Error 1
make[2]: Leaving directory `/export/home/jaap/Downloads/sage-4.3.2.alpha0/spkg/build/singular-3-1-0-9-20100125/src/factory'
make[1]: *** [install] Error 1
make[1]: Leaving directory `/export/home/jaap/Downloads/sage-4.3.2.alpha0/spkg/build/singular-3-1-0-9-20100125/src'
make: *** [/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/bin/Singular-3-1-0] Error 2
Unable to build Singular.

real    0m24.240s
user    0m12.235s
sys 0m9.934s
sage: An error occurred while installing singular-3-1-0-9-20100125

Maybe it will install a some FLAGS are set globally (-m64) in this case).

As Open Solaris is not a supported platform I'll not interfere with the positive review.

Jaap

[aghitza]

comment:16

Jaap,

Did the previous Singular spkg build properly on Open Solaris? If so, I wonder if it's just a problem in spkg-install, or a more serious issue with Singular itself.

[jaapspies]

comment:17

If CFLAGS and friends are set tight. Building on Open Solaris fails on

gcc -g -pg -O3 -I. -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include -O2 -g -m64 -I/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/include -DHAVE_CONFIG_H -c omBinPage.c -o omBinPage.op
/var/tmp//ccciayz2.s: Assembler messages:
/var/tmp//ccciayz2.s:31: Error: junk `@' after expression
/var/tmp//ccciayz2.s:110: Error: junk `@' after expression
/var/tmp//ccciayz2.s:183: Error: junk `@' after expression
/var/tmp//ccciayz2.s:270: Error: junk `@' after expression
/var/tmp//ccciayz2.s:477: Error: junk `@' after expression
/var/tmp//ccciayz2.s:805: Error: junk `@' after expression
/var/tmp//ccciayz2.s:1240: Error: junk `@' after expression
/var/tmp//ccciayz2.s:1718: Error: junk `@' after expression
/var/tmp//ccciayz2.s:2041: Error: junk `@' after expression
make[2]: *** [omBinPage.op] Error 1
make[2]: Leaving directory `/export/home/jaap/Downloads/sage-4.3.2.alpha0/spkg/build/singular-3-1-0-9-20100125/src/omalloc'
make[1]: *** [install] Error 1
make[1]: Leaving directory `/export/home/jaap/Downloads/sage-4.3.2.alpha0/spkg/build/singular-3-1-0-9-20100125/src'
make: *** [/export/home/jaap/Downloads/sage-4.3.2.alpha0/local/bin/Singular-3-1-0] Error 2
Unable to build Singular.

Actually this is the same failure I had before.

What is the use of -gp in building singular for sage?

If I remove -gp in Makefile and Makefile.in all, over singular builds on Open Solaris x64 64 bit

Jaap

[jaapspies]

comment:18

Replying to @aghitza:

Jaap,

Did the previous Singular spkg build properly on Open Solaris? If so, I wonder if it's just a problem in spkg-install, or a more serious issue with Singular itself.

The -gp flag leads to problems on Open Solaris.

Jaap

[malb]

comment:19

man gcc says:

-pg Generate extra code to write profile information suitable for the analysis program gprof.  You must use this option when compiling the source files you want data about, and you must also use it when linking.

so this shouldn't be used by default anyway.

[jaapspies]

comment:20

Replying to @malb:

man gcc says:

-pg Generate extra code to write profile information suitable for the analysis program gprof.  You must use this option when compiling the source files you want data about, and you must also use it when linking.

so this shouldn't be used by default anyway.

+1

Thanks.

Jaap

[aghitza]

comment:21

Martin, do you want to remove this option in the spkg? I'd be happy to do some test builds to make sure that it doesn't break anything on other architectures. But if the fix is this simple to make it build on Open Solaris, we might as well do it now.

[jaapspies]

comment:22

Replying to @aghitza:

Martin, do you want to remove this option in the spkg? I'd be happy to do some test builds to make sure that it doesn't break anything on other architectures. But if the fix is this simple to make it build on Open Solaris, we might as well do it now.

I think it need upstream work. In the former release it was in patches. Now it is in source :(

Removing -gp will not break anything in sage. We do no profiling!

Jaap

[malb]

comment:23

Looking at Jaap's error again it comes from omalloc. For omalloc all targets are built which are

• libomalloc (normal)
• libomalloc_ndebug (no debugging symbols)
• libomalloc_p (for profiling)

We shouldn't mess with the -pg switch but should convince omalloc's autotools not to build the profiling version on top of the other two.

[jaapspies]

comment:24

One more comment. Only developers of a program or other software can put interest in profiling there product.

So once more: let's ask upstream to remove this option.

For now it is an option to introduce some patches again in the spkg-install :(

Shall we set status to needs work?

Jaap

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:25

I would also add I think adding profiling information by default is a bad idea.

• Adding profiling information will slow the code quite considerably.
• The -pg option is not portable, so it would hamper efforts to build with a non-GNU compiler.
• The purposely add an option which is breaking a build on Open Solaris, when an Open Solaris port is in progress, seems a bad idea.
• As Jaap points out, only developers want this information - not end users.

I would be tempted to set it to 'needs work', but feel I can't really do so, since Open Solaris is not supported on Sage. So the fact an option breaks on an unsupported operating system is not a sufficiently good reason for rejecting it. But I would urge the person that created the ticket, or the reviewer to do so. (BTW, the Reviewer field is not filled in).

Dave

[malb]

comment:26

Guys,

no one wants to add -pg to the default options for Singular and no one did! Most Singular libraries are built in several flavours as explained above but not all of them are used of course. It's the build of an unused library that is going wrong.

There is a discussion on this issue on [libsingular-devel]:

http://groups.google.com/group/libsingular-devel/browse_thread/thread/43d157fe730e8798

where Hans Schönemann suggested to patch omalloc. It is still puzzling why the build breaks now since IIRC nothing changed in that area.

[malb]

comment:27

Is anybody working on patching the Makefiles? It might take a few days before I'll get around to do it.

[jaapspies]

comment:28

Replying to @malb:

Is anybody working on patching the Makefiles? It might take a few days before I'll get around to do it.

Are you getting around to do it?

Jaap

[malb]

comment:29

Not yet, feel free to go ahead ;)

[malb]

comment:30

I thought about this patch-all-Makefiles approach some more. I came to the conclusion that it would be a bad idea. Jaap's GCC or the Singular's configuration script is broken. GCC accepts -pg and that GCC what is called. That it fails hints at either a bug in the OpenSolaris GCC or at a problem choosing the right assembler or linker.

Patching all Makefiles would have serious repercussions: this patch would never be accepted upstream and thus we would have to go through a very tedious and error-prone process of patching every Makefile with every SPKG update. Since I am right now the only person who does these updates at the moment, I think I refuse to increase my workload like this based on one faulty setup.

Note that I am happy to investigate and push upstream any fix which triggers this error in the configuration script. I want Sage to build and work on OpenSolaris and Solaris. It's just this approach which I deem incorrect.

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:31

If there are a lot of Makefiles, then patching would be tedious and require more work with upgrades. However, it should not be too hard to create a small script which patches the Makefiles and call that script from spkg-install.

I've not checked this on the Singular sources, but this sed script, which I called 'nopg' could be used to replace ' -pg ' with nothing

#!/bin/sh 
sed 's/ -pg //g' $1 > /tmp/Makefile-without-pg-option.$$
mv /tmp/Makefile-without-pg-option.$$  $1

then in spkg-install have:

find src -name Makefile -exec /path/to/nopg {} \;

I tried it on a gcc source distribution, which has loads of 'configure' scripts, replacing 'echo' with 'zzzzzzzzzzzzzzzzz' and sure enough, all the configure script have strings of zzzzz's in them.

It would be necessary to check the files have the required changes, and no other changes, but that should be a simple solution.

Dave

Dave

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:32

PS, the substitution would be from ' -pg ' to nothing at all, so it would not break if files in the makefile have -pg in their names.

[malb]

comment:33

That indeed might be a viable solution. However, we should still investigate what exactly breaks Jaap's setup instead of working around the issue which we don't understand.

[malb]

comment:34

A new SPKG is available at #8228, you will need the attached patch for this ticket to make it work.

[malb]

comment:35

I vote to review 3-1-0-9 without considering Jaap's bug and then opening a new ticket where Jaap gives more details so that we can address the bug. Right now, Sage crashes in a fairly basic situation and we thus should upgrade.

[aghitza]

comment:36

Martin, this is probably not the question you want to hear, but: would it be worth it to go straight to 3-1-1, since it was just released?

[malb]

comment:37

Replying to @aghitza:

Martin, this is probably not the question you want to hear, but: would it be worth it to go straight to 3-1-1, since it was just released?

I'll make you a deal: if you review it I'll make a Singular-3-1-1 SPKG :)

[aghitza]

comment:38

Fine, I'll take that. I should be able to do it this weekend.

[malb]

Attachment: singular-3-1-1-0.patch.gz

[malb]

comment:39

A new SPKG is available at

http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-1-0-20100305.spkg

[malb]

comment:40

There are doctest failures in the scheme subdirectory. I reported one bug upstream already.

[malb]

comment:41

Doctest failures:

The following tests failed:

        sage -t  devel/sage/sage/schemes/plane_curves/projective_curve.py # 14 doctests failed
        sage -t  devel/sage/sage/schemes/plane_curves/curve.py # 6 doctests failed
        sage -t  devel/sage/sage/schemes/plane_curves/constructor.py # 6 doctests failed

[malb]

comment:42

After fixing a typo in a Singular routine

http://groups.google.com/group/libsingular-devel/browse_frm/thread/de4c7123c71f2150

I get a bit further, but I need some help now:

I assume '32' is wrong in the failure below?

sage -t  "devel/sage/sage/schemes/plane_curves/constructor.py"
**********************************************************************
File "/mnt/usb1/scratch/malb/sage-4.3.3/devel/sage/sage/schemes/plane_curves/constructor.py", line 72:
    sage: C.genus()
Expected:
    0
Got:
    32

Am I correct that the following is also wrong?

sage -t  "devel/sage/sage/schemes/plane_curves/curve.py"
   ? ideal defines not a curve
   ? leaving normal.lib::genus
**********************************************************************
File "/mnt/usb1/scratch/malb/sage-4.3.3/devel/sage/sage/schemes/plane_curves/curve.py", line 79:
    sage: C.geometric_genus()

[malb]

comment:43

The different behaviour in pure Singular:

> ring r = 7,(x,y),dp;
> LIB("normal.lib");
> ideal i = x^10 + x^3 + y^2;
> genus(i);
32

[aghitza]

comment:44

Martin, I just tried the new spkg but I'm running into trouble very quickly: after doing sage -f ..., I'm getting errors when doing sage -b

This is under sage-4.3.4.alpha0, and I got it now on two machines (including sage.math)

[malb]

comment:45

Did you apply singular-3-1-1-0.patch?

[aghitza]

comment:46

Hmmm that's pretty embarrassing.

So it builds fine now. Note that applying the patch to either 4.3.3 or 4.3.4.alpha0 results in one rejection. It is actually harmless, but it might be good to rebase anyway.

I'm pretty sure that the (geometric) genus should be 0 in the doctest that fails claiming it's 32. (In particular, both Macaulay2 and Magma say 0.)

I guess this should be reported upstream, since they know better what they've changed since 3-1-0-9 in normal.lib.

[malb]

Upstream: Fixed upstream, in a later stable release.

[malb]

comment:47

The bug I isolated above will be fixed in the next release:

Hi Martin,

Thank you for reporting this bug.
It will be corrected in the next release.

Gert-Martin

2010/3/8 Martin Albrecht <malb@informatik.uni-bremen.de>:
> Hi,
>
> the following output is wrong in 3-1-1-0 and was correct in the previous
> version:
>
>> ring r = 7,(x,y),dp;
>> LIB("normal.lib");
>> ideal i = x^10 + x^3 + y^2;
>> genus(i);
> 32 //should be 0

We should try to make sure that we report all bugs (I think there is another one were curves are considered not to be curves) before they cut the new release.

[aghitza]

comment:48

We should try to make sure that we report all bugs (I think there is another one were curves are considered not to be curves) before they cut the new release.

Yep. If I have some time (cross fingers) I'll try to test 3-1-1-0 with a few more examples and see if I catch anything else. Any idea what the timeline is for the new release?

[malb]

comment:49

No idea, but I just reported another bug upstream:

> ring r = 5,(x,y,z),dp;
> ideal i = -x^3 + y^2*z - 2*x*z^2 + y*z^2;
> LIB("normal.lib");
> genus(i);
   ? ideal defines not a curve
   ? leaving normal.lib::genus

Another thing we might want to tackle is that it seems that our libsingular interface isn't very good at recovering from errors. I'm not sure how to address this; I guess I'll ask upstream some time.