Upgrade to Singular-4.x.x

[jdemeyer]

Lots of stuff has changed in Singular 4. This is a big update.

Build system completely changed, and a lot of internal stuff.

Original upstream tarball at www.mathematik.uni-kl.de/ftp/pub/Math/Singular/SOURCES/4-0-3/: singular-4.0.3p3 dot tar dot gz

Modified tarball made with spkg-src at:

• http://www.lmona.de/files/sage/singular-4.0.3p3.tar.bz2

This corresponds to commit a070b84d whose message tells 4.0.3p3 and on which this branch was based:

• https://github.com/Singular/Sources/commit/a070b84d59be835a2946d1386edd3221d80f3fe7

You can also make your own tarball based on this commit or another one with

env SINGULAR_GIT_COMMIT=a070b84d ./sage --sh build/pkgs/singular/spkg-src
./sage --package fix-checksum singular

This script is loosely based on the make_tar.sh script from upstream.

Depends on #17635

CC: @jpflori @burcin @sagetrac-jakobkroeker @kiwifb @malb @mkoeppe @vbraun @nthiery @bhutz @miguelmarco @sagetrac-gjorgenson

Component: packages: standard

Author: Jakob Kroeker, Jean-Pierre Flori, Jeroen Demeyer, John Perry, François Bissey, Leif Leonhardy, Dima Pasechnik

Branch: f1a0dcc

Reviewer: François Bissey, Jeroen Demeyer, Ben Hutz, Leif Leonhardy, Dima Pasechnik, Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/17254

[kiwifb]

comment:37

Did you mean local/var/tmp/sage/build/singular-4.0.1p1.p0/src/Singular/ the latest bit should be stripped by the -p1 option to patch.

[johnperry-math]

comment:38

The patch has latest in it; that's why I wrote that in there, so that is what I mean. Is your meaning that latest should not be there? That directory doesn't exist when I look through the paths.

[kiwifb]

comment:39

It is expected for that directory not to exist for you. Whoever wrote the patch was working in a folder named latest but it is their own naming. It doesn't matter anyway. Let me explain to you how the -p option of patch works.

Your patch has a header that helps you locate the file to patch:

--- latest/Singular/run.c       2014-11-19 14:06:05.000000000 +0100
+++ latest/Singular/run.c       2015-01-16 09:32:45.771298300 +0100

patch < mypatch.patch or equivalently patch -p0< mypatch.patch will look for the file latest/Singular/run.c relative to where patch is executed. -pN will strip the N first folders in front of the file name so

patch -p1< mypatch.patch will look for Singular/run.c. In spkg-install for singular, patches are applied with patch -p1 <"$patch" so you shouldn't care about latest it is discarded. Look for local/var/tmp/sage/build/singular-4.0.1p1.p0/src/Singular/run.c OK?

[johnperry-math]

comment:40

I think you misunderstand.

Replying to @kiwifb:

It is expected for that directory not to exist for you.

Why not? That directory does exist for me. The problem is that the file run.c does not exist in it.

Look for local/var/tmp/sage/build/singular-4.0.1p1.p0/src/Singular/run.c OK?

There is no such directory. I can get to local/var/tmp/sage/build/singular-4.0.1p1.p0/src/ but the only directories in there are latest and shared.

[kiwifb]

comment:41

OK you seem to be right. The tarball has been packaged strangely and I am not sure I have the right spkg-install. It is a cygwin patch and cygwin support in singular seems to have changed considerably. I would just remove the patch for now, it doesn't/cannot apply anywway any new fix for cygwin would have to be done from scratch by someone with a cygwin system.

[johnperry-math]

comment:42

How do I remove it? If I move it & make anew, the file reappears.

[kiwifb]

comment:43

git rm build/pkgs/singular/patches/stricmp.patch?

[johnperry-math]

comment:44

Thanks, that worked. Now I get failure on a line in ring.pyx,

    if is_64_bit:

Namely, Cython (?) claims this is undefined. I guess this is a change in Cython, since that isn't introduced by this patch. I found online another way to test this (sys.maxsize > 2**32) so I'm trying with that. Found another issue, but that was easy. Proceeding with fingers crossed... it's still compiling, which is already an progress...!

[johnperry-math]

comment:45

It seems to have built, but Sage doesn't run. Here's the last few lines of the compilation (sorry for so much, but I wanted to show what it was saying at the end of what looks like a successful build, too):

...
byte-compiling /Applications/sage-6.7/local/lib/python2.7/site-packages/sage_setup/find.py to find.pyc
byte-compiling /Applications/sage-6.7/local/lib/python2.7/site-packages/sage_setup/optional_extension.py to optional_extension.pyc
running install_egg_info
Removing /Applications/sage-6.7/local/lib/python2.7/site-packages/sage-6.8.beta0-py2.7.egg-info
Writing /Applications/sage-6.7/local/lib/python2.7/site-packages/sage-6.8.beta0-py2.7.egg-info

real    0m25.849s
user    0m11.855s
sys 0m3.847s
[ -f local/etc/sage-started.txt ] || local/bin/sage-starts
build/pipestatus "./sage --docbuild --no-pdf-links all html  2>&1" "tee -a logs/dochtml.log"
Deleting empty directory /Applications/sage-6.7/src/doc/common/static
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/combinat/static
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/combinat/templates
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/polynomial_rings/static
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/polynomial_rings/templates
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/repl/static
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/repl/templates
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/tensor_free_modules/static
Deleting empty directory /Applications/sage-6.7/src/doc/en/reference/tensor_free_modules/templates
Deleting empty directory /Applications/sage-6.7/src/doc/output/inventory/en/reference/combinat
Traceback (most recent call last):
  File "/Applications/sage-6.7/src/doc/common/builder.py", line 1619, in <module>
    import sage.all
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/all.py", line 98, in <module>
    from sage.rings.all      import *
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/all.py", line 68, in <module>
    from number_field.all import *
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/number_field/all.py", line 7, in <module>
    from totallyreal import enumerate_totallyreal_fields_prim
  File "sage/rings/number_field/totallyreal_data.pxd", line 16, in init sage.rings.number_field.totallyreal (build/cythonized/sage/rings/number_field/totallyreal.c:10339)
  File "sage/rings/number_field/totallyreal_data.pyx", line 39, in init sage.rings.number_field.totallyreal_data (build/cythonized/sage/rings/number_field/totallyreal_data.c:11133)
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_ring_constructor.py", line 465, in PolynomialRing
    R = _single_variate(base_ring, name, sparse, implementation)
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_ring_constructor.py", line 539, in _single_variate
    R = m.PolynomialRing_integral_domain(base_ring, name, sparse, implementation)
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_ring.py", line 1544, in __init__
    sparse=sparse, element_class=element_class)
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_ring.py", line 1451, in __init__
    sparse=sparse, element_class=element_class, category=category)
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_ring.py", line 305, in __init__
    from sage.matrix.matrix_space import is_MatrixSpace
  File "/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/matrix/matrix_space.py", line 57, in <module>
    import matrix_mpolynomial_dense
ImportError: dlopen(/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/matrix/matrix_mpolynomial_dense.so, 2): Symbol not found: __Z17initCanonicalFormv
  Referenced from: /Applications/sage-6.7/local/lib/python2.7/site-packages/sage/matrix/matrix_mpolynomial_dense.so
  Expected in: flat namespace
 in /Applications/sage-6.7/local/lib/python2.7/site-packages/sage/matrix/matrix_mpolynomial_dense.so
make: *** [doc-html] Error 1

[kiwifb]

comment:46

It appears that it should be generated by initCanonicalForm but no functions of that name is to be found in the source code I have for singular 4.0.2 here and I am expecting it to be the same for you. I think you may have old singular headers on your system that are interfering (factory.h) we'll have to develop a proper upgrade procedure to remove singular 3.1.x stuff before installing 4.x. That or the singular 4.x headers have not been installed properly. local/include/factory/factory/factory.h and local/include/factory/factory.h both have this function in singular 3.1.7 but it shouldn't be present in the equivalent headers from singular 4.x.

[johnperry-math]

comment:47

I don't seem to have a local/include/factory at all...

[kiwifb]

comment:48

I really have to inspect the spkg. I think I went overboard with the factory in the path. Should have been local/include/factory.h and local/include/factory/factory.h. Do you have a factory.h and does it contains initCanonicalForm?

[johnperry-math]

comment:49

make distclean then make seems to have fixed my problems. I am apparently running Singular 4, albeit with the changes I mentioned earlier. At startup, I see the following:

ring with rational coefficient field created
_ring.ShortOut 1
_ring.N 2
polynomial ring over integers created
_ring.ShortOut 1
_ring.N 2
ring with rational coefficient field created
_ring.ShortOut 1
_ring.N 2

I found that in the source code, so I'm guessing this was put in to test.

[johnperry-math]

comment:50

Well, this is bad.

sage: R=PolynomialRing(GF(32003),'x',7)
sage: I = sage.rings.ideal.Cyclic(R,7).homogenize()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-59ae6a3e91c7> in <module>()
----> 1 I = sage.rings.ideal.Cyclic(R,Integer(7)).homogenize()

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/rings/ideal.pyc in Cyclic(R, n, homog, singular)
   1607         n = R.ngens()
   1608 
-> 1609     singular.lib("poly")
   1610     R2 = R.change_ring(RationalField())
   1611     R2._singular_().set_ring()

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/singular.pyc in lib(self, lib, reload)
    783         if not reload and lib in self.__libs:
    784             return
--> 785         self.eval('LIB "%s"'%lib)
    786         self.__libs.append(lib)
    787 

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/singular.pyc in eval(self, x, allow_semicolon, strip, **kwds)
    585             x += ';'
    586 
--> 587         s = Expect.eval(self, x, **kwds)
    588 
    589         if s.find("error") != -1 or s.find("Segment fault") != -1:

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/expect.pyc in eval(self, code, strip, synchronize, locals, allow_use_file, split_lines, **kwds)
   1220                 elif split_lines:
   1221                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1222                                         for L in code.split('\n') if L != ''])
   1223                 else:
   1224                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/expect.pyc in _eval_line(self, line, allow_use_file, wait_for_prompt, restart_if_needed)
    817         try:
    818             if self._expect is None:
--> 819                 self._start()
    820             E = self._expect
    821             try:

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/singular.pyc in _start(self, alt_message)
    410         """
    411         self.__libs = []
--> 412         Expect._start(self, alt_message)
    413         # Load some standard libraries.
    414         self.lib('general')   # assumed loaded by misc/constants.py

/Applications/sage-6.7/local/lib/python2.7/site-packages/sage/interfaces/expect.pyc in _start(self, alt_message, block_during_init)
    427                 # Change pexpect errors to RuntimeError
    428                 raise RuntimeError("unable to start %s because the command %r failed: %s\n%s" %
--> 429                         (self.name(), cmd, e, self._install_hints()))
    430         except BaseException:
    431             self._expect = None

RuntimeError: unable to start singular because the command 'Singular -t --ticks-per-sec 1000' failed: The command was not found or was not executable: Singular.

[johnperry-math]

comment:51

For what it's worth, lots of other things work.: I can, for instance, compute an ideal's Gröbner basis, its Hilbert [numerator, series, polynomial], and so forth.

[johnperry-math]

comment:52

Indeed, there is no Singular binary in local/bin. I presume this is the problem, though I have no idea what became of it.

[johnperry-math]

comment:53

I'm guessing that failure to install the singular binary is because of a symlink: local/bin/singular' points tolocal/bin/Singularand removing the old (3.x)Singularwasn't enough to allow the new one in. I removedlocal/bin/singular` and it's now working.

This may be an issue with Apple's file system, because after installing the new Singular I wasn't allowed to create the symlink.

[jdemeyer]

comment:54

is_64_bit is deprecated. In Cython, check sizeof(the_type_that_you_care_about) instead.

Do not use sys.maxsize, that's much slower.

[kiwifb]

comment:55

Replying to @johnperry-math:

I'm guessing that failure to install the singular binary is because of a symlink: local/bin/singular' points tolocal/bin/Singularand removing the old (3.x)Singularwasn't enough to allow the new one in. I removedlocal/bin/singular` and it's now working.

This may be an issue with Apple's file system, because after installing the new Singular I wasn't allowed to create the symlink.

Yes you may have seen another ticket in which I demonstrated that something like Singular was really singular. OS X file system is not case sensitive but it gives the impression of being one.

[johnperry-math]

comment:56

Replying to @johnperry-math:

I just applied this to 6.4.1 and encountered the message,

fatal error: Singular/libsingular.h: No such file or directory

while compiling

build/cythonized/sage/algebras/letterplace/free_algebra_element_letterplace.cpp

Any ideas what causes this?

It's now certain that this, too, was caused by the failure to install Singular because singular exists in the directory (what can I say, it's a Mac). Removing that link and reinstalling the spkg fixed all these problems.

Meanwhile, I discovered some memory-related bugs in the patch. For instance, ring.pyx has the line

assert( exponent == base_ring.degree() )

which causes a failure. I'm not sure why those should be equal (the test I used had Zmod(2^3) where exponent==3 and base_ring.degree()==1), but commenting it out leads to a failure on these lines:

_param.GFChar     = ch;
_param.GFDegree   = base_ring.degree();
_param.GFPar_name = omStrDup(base_ring.gen());   

because _param has not been initialized. Adding

_param = <GFInfo *>omAlloc(sizeof(GFInfo));

before the access gives me a ring. However,

1. I'm not sure whether omAlloc or one of the other omAlloc* commands is more appropriate (e.g., omAlloc0). Anyone know?
2. I suppose this has to be freed somewhere... I know there's an obvious-sounding place, so I'll find it.

There are other places this change has to occur. One more question:

• A few lines before the ones I cited above is the test for 64-bits. I've switched to jdemeyer's test (sizeof(long)>4). However, the code in the patch is exactly the same for both 32- and 64-bit. Is there a recommendation for what to do here? I'm not sufficiently familiar with this to know what to do.

[johnperry-math]

comment:57

Replying to @johnperry-math:

Meanwhile, I discovered some memory-related bugs in the patch. For instance, ring.pyx has the line

assert( exponent == base_ring.degree() )

which causes a failure. I'm not sure why those should be equal (the test I used had Zmod(2^3) where exponent==3 and base_ring.degree()==1)...

To wit, the source code for base_ring.degree() is, in total,

    def degree(self):
        """
        Return 1.

        EXAMPLE::

            sage: R = Integers(12345678900)
            sage: R.degree()
            1
        """
        return integer.Integer(1)

I don't know what is meant by the degree of a ring here, so I'll need guidance. For now I've commented it out.

[johnperry-math]

Changed branch from u/jakobkroeker/ticket.17254.squashed to u/john_perry/ticket.17254.squashed

[johnperry-math]

Changed commit from 3a512cd to 1f4ebcb

[johnperry-math]

comment:59

Pushed code should build on 6.7. Mac users may have problems; if so, delete local/bin/singular and then sage-spkg singular; it should work after that. (In fact, I have to do it again myself now.) I have to run to the train (leaving SD) or I'd do it before leaving.

---

New commits:

bdb0851	first attempt to resolve
1f4ebcb	fix for _param and discrepancy between exponent and degree

[johnperry-math]

comment:60

Forgot to remove that bad patch; that will come in a subsequent push. Still have the problem with local/bin/Singular because of the local symlink; it's a problem with the package itself, because I deleted both symlink and local copy before running the install. I don't know how to fix that, so advice would be appreciated.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1f4ebcb to 9aa2456

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9aa2456

delete stricmp.patch

[jdemeyer]

comment:62

Replying to @johnperry-math:

However, the code in the patch is exactly the same for both 32- and 64-bit. Is there a recommendation for what to do here?

Obviously, replace

if condition:
    X
else:
    X

by

X

[jdemeyer]

comment:63

Replying to @johnperry-math:

• A few lines before the ones I cited above is the test for 64-bits. I've switched to jdemeyer's test (sizeof(long)>4). However, the code in the patch is exactly the same for both 32- and 64-bit. Is there a recommendation for what to do here? I'm not sufficiently familiar with this to know what to do.

Look at the diff for the old corresponding code: this doesn't have the branch.

[jdemeyer]

comment:64

I would also recommend you to keep the logic of the old code where applicable. For example, the old code started with

if ch.is_power_of(2):
exponent = ch.nbits() -1
    # it seems Singular uses ints somewhere
    # internally, cf. #6051 (Sage) and #138 (Singular)
    if exponent <= 30:

and I see no reason to change this.

[fchapoton]

Changed branch from u/john_perry/ticket.17254.squashed to public/ticket/17254

[fchapoton]

Changed commit from 9aa2456 to ab68eb0

[fchapoton]

comment:65

naive rebase.

The spkg is no longer available from boxen..

---

New commits:

ab68eb0

Merge branch 'u/john_perry/ticket.17254.squashed' into 6.9.b0

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

comment:66

The [singular 4.0.1] spkg is no longer available from boxen..

That is unfortunate. @jdemeyer/@jpflori: could you upload the singular 4.0.1.p1 tarball to somewhere else?

naive rebase. [by chapoton] Conflicts: src/sage/libs/singular/singular-cdefs.pxi

the file src/sage/libs/singular/singular-cdefs.pxi is even missing in the ticket branch public/ticket/17254

@fchapoton Did you set the ticket branch to broken public/ticket/17254 instead of leaving it 'u/john_perry/ticket.17254.squashed' by accident?

[jpflori]

comment:67

It seems I don't have access to boxen anymore...

[jpflori]

comment:68

And I don't have the p1 tarball Jeroen posted.

[kiwifb]

comment:69

I have a tarball answering that description. Up on the lmona.de server....

http://www.lmona.de/files/sage/singular-4.0.1p1.tar.bz2

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

Description changed:

--- 
+++ 
@@ -1,3 +1,3 @@
 **Lots** of stuff has changed in Singular 4. But now the attached branch is almost working.

-Repackaged upstream tarball: [http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-4.0.1p1.tar.bz2](http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-4.0.1p1.tar.bz2)
+Repackaged upstream tarball: [http://www.lmona.de/files/sage/singular-4.0.1p1.tar.bz2](http://www.lmona.de/files/sage/singular-4.0.1p1.tar.bz2)

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

Changed commit from ab68eb0 to aa40c90

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

comment:71

bumped to recent sage dvelopment version.

The build may fail with 'undefined ..' error again. Probably this can be fixed with ./sage -ba; if not, make distclean-bazooka should work...

---

New commits:

8e259d1	merged with recent develop
aa40c90	check for local tarball

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

Changed branch from public/ticket/17254 to u/jakobkroeker/ticket.17254.squashed

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

comment:72

@fchapoton

naive rebase.

except for the issue with src/sage/libs/singular/singular-cdefs.pxi, how did you do the rebase? Did you a rebase by hand, or did you create a patch after merge and then apply the patch to 6.9.b0?

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from aa40c90 to 72ef927

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

72ef927

squashed singular upgrade changes in one commit

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 72ef927 to 4adc75d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4adc75d

squashed singular upgrade changes in one commit

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f492ef2

squashed singular upgrade changes in one commit

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4adc75d to f492ef2

[ea1d0bf8-c27a-4548-8cb7-de0b1d02441a]

comment:76

@johnperry-math thanks for the bugfix!

I am willing to work on this

somehow I overlooked your messages in my emails in the past.Are you still available for help? Maybe we could work together on the remaining issues? (skype?)

An example of creating extension fields can be found in Singulars libpolys/tests/polys_test.h :: test_Q_Ext_a()) while the basic data structure idea is sketched in libpolys/polys/ext_fields/algext.h

[johnperry-math]

comment:77

Replying to @sagetrac-jakobkroeker:

@johnperry-math somehow I overlooked your messages in my emails in the past.Are you still available for help? Maybe we could work together on the remaining issues? (skype?)

I'm willing to collaborate on it, but I've had a more pressing project which has consumed my attention/free time. I thought I'd be done with it months ago :-) so I can't promise to be very helpful and/or focus on this at the moment, but I do hope to be available from time to time, until the current project passes in which case I could focus on this a little bit.