Upgrade of p_group_cohomology spkg

simon-king-jena commented 9 years ago

In the past, p_group_cohomology-2.1.4 was an optional spkg. Sage dropped support for old style spkgs, and thus only an experimental spkg remained.

The purpose of this ticket is to provide a new group cohomology package for Sage. I suggest to make it optional (not experimental).

In the experimental version 2.1.5, I have improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

Package structure

Version 3.0 is refactored as follows:

It depends on the optional meataxe package (see #24359), which is based on a fork of MeatAxe.
The spkg-install script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
The C code written by David Green has become an autotoolized shared library.
Gap and Singular code is put into SAGE_SHARE/, where GAP and Singular libraries belong.
The rest of the package is formed by python and cython code, and uses pip for installation.
The documentation is built in an improved way.

Further changes

The custom logging in the old spkg is replaced by Python's logging module.
The tests avoid side-effects by using a reset function. Hence, no custom doc tester is needed: sage -t is enough.
In the previous spkg, I tried to be as backward compatible as possible. But now I'm dropping support for very old Singular versions.
Some experimental options for the cohomology computation are removed.

Many tests have changed, since the ring presentations of the computed cohomology rings, specifically for non-primepower groups, have changed. I do not fully understand why they changed. but it seems that it is a consequence of changes in Singular. The changes did (of course...) not affect the isomorphism types of the rings.

Upstream tar ball

https://users.fmi.uni-jena.de/cohomology/p_group_cohomology-3.0.tar.xz

Upstream: None of the above - read trac for reasoning.

CC: @jhpalmieri @vbraun @jdemeyer @haraldschilly @frederichan-IMJPRG david.green@uni-jena.de

Component: packages: optional

Keywords: group cohomology

Author: Simon King

Branch: 6576915

Reviewer: Travis Scrimshaw, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/18514

simon-king-jena commented 9 years ago

comment:1

For me, all but two tests fail. Both tests fail with "connection timed out" during access to a web data base, and both work fine when done in an interactive session.

Has something in the test framework changed, so that web access via urllib2 fails during tests?

jdemeyer commented 9 years ago

comment:2

I see you are compiling some extensions with the csage library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage only if it actually exists.

simon-king-jena commented 9 years ago

comment:3

Replying to @jdemeyer:

I see you are compiling some extensions with the csage library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage only if it actually exists.

OK. So I should see what I actually use of csage.

jdemeyer commented 9 years ago

comment:4

The sig_on() framework was in csage until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage if the Sage version is <= 6.7

simon-king-jena commented 9 years ago

comment:5

Replying to @jdemeyer:

The sig_on() framework was in csage until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage if the Sage version is <= 6.7

OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off, or would this just work automatically with the new Cython version?

jdemeyer commented 9 years ago

comment:6

Replying to @simon-king-jena:

OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off, or would this just work automatically with the new Cython version?

You just include sage/ext/interrupt.pxi, that doesn't change.

The main difference is that you no longer need to add csage as library. Currently, it doesn't hurt if you add the library anyway, but in the future, csage might not exist anymore.

And there is one additional caveat which is relevant for your package: since #18027, you must include interrupt.pxi only in .pyx files and not .pxd files (unless you really know what you're doing).

jdemeyer commented 9 years ago

comment:7

On the topic of .pxd files, this is what they should contain:

The API of your module, that is all cdef classes and c(p)def functions that you want to export for use in other modules (in the same package or by other packages).
If your module provides bindings to some library (the mtx module in your package for example), then add the declarations of the library functions in the .pxd file.
Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from RingElement. In your .pxd file, you want to write cdef class MySpecialElement(RingElement). This makes RingElement a dependency, so you need to write from ... cimport RingElement in the .pxd file.

And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx file. This is important in order to minimize dependencies.

NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd file. In Sage, we usually put the library bindings in sage/libs, look at sage/libs/gmp for a good example.

simon-king-jena commented 9 years ago

comment:8

Replying to @jdemeyer:

On the topic of .pxd files, this is what they should contain:

The API of your module, that is all cdef classes and c(p)def functions that you want to export for use in other modules (in the same package or by other packages).

If your module provides bindings to some library (the mtx module in your package for example), then add the declarations of the library functions in the .pxd file.

Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from RingElement. In your .pxd file, you want to write cdef class MySpecialElement(RingElement). This makes RingElement a dependency, so you need to write from ... cimport RingElement in the .pxd file.

And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx file. This is important in order to minimize dependencies.

OK. That's something that I should change. Actually, until not so long ago, I thought that cimports were only possible in pxd files.

NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd file. In Sage, we usually put the library bindings in sage/libs, look at sage/libs/gmp for a good example.

Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe bindings, say, mtx_lib.pxd, which is then cimported in mtx.pxd?

jdemeyer commented 9 years ago

comment:9

Replying to @simon-king-jena:

Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe bindings, say, mtx_lib.pxd, which is then cimported in mtx.pxd?

I cannot really speak about this specific case, but it certainly would make sense to split it up.

simon-king-jena commented 9 years ago

Work Issues: Cope with yet another backward incompatible change of Sage

simon-king-jena commented 9 years ago

comment:11

I just found that the package won't build with the latest beta of Sage. Reason:

pGroupCohomology/cohomology.c:248:36: fatal error: sage/ext/interrupt/pxi.h: No such file or directory
 #include "sage/ext/interrupt/pxi.h"
                                    ^
compilation terminated.
error: command 'gcc' failed with exit status 1

So, can you please tell me how to use interrupts, after all these backward incompatible changes? It really sucks.

simon-king-jena commented 9 years ago

comment:12

Good news! On my machine, I did some refactoring of code according to the hints you gave me---and that did the trick.

My plan is to continue refactoring, taking into account what you said about where to import stuff, and then push a new package version to the official page.

simon-king-jena commented 9 years ago

comment:13

Bad news. It builds, but I can't import it. Which brings up the question again why there are all these incompatible changes in Sage. They didn't use to occur so frequently in earlier versions of Sage, IIRC.

simon-king-jena commented 9 years ago

comment:14

I got it fixed. So, you'll find the updated version of p_group_cohomology-2.1.5 on my web page.

Some of your concerns about importing/including stuff are addressed in the new version: I moved some of it from .pxd files into .pyx files. What I did not do is to split off library bindings and put them into separate .pxd files. That would still make sense, but for now I put the ticket back to "needs review".

simon-king-jena commented 9 years ago

Changed work issues from Cope with yet another backward incompatible change of Sage to none

simon-king-jena commented 9 years ago

comment:15

Oh, and I also created a new module pGroupCohomology.auxiliaries, in which I've put some auxiliary functions that have previously been in pGroupCohomology.resolution.

frederichan-IMJPRG commented 9 years ago

comment:16

I have a built failure for sage -i upstream/p_group_cohomology-2.1.5.spkg with sage 6.8beta3+trac18666 (but success with master)

gcc -pthread -shared -L/home/fred/dev/sage/local/lib build/temp.linux-x86_64-2.7/pGroupCohomology/resolution.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufloesung.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufnahme.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/djgerr.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/fileplus.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/nBuchberger.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pgroup.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pincl.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/slice.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/urbild.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/uvr.o -L/home/fred/dev/sage/local/lib -lmtx -lpython2.7 -o build/lib.linux-x86_64-2.7/pGroupCohomology/resolution.so
cythoning pGroupCohomology/cohomology.pyx to pGroupCohomology/cohomology.c
building 'pGroupCohomology.cohomology' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/fred/dev/sage/src/c_lib/include -I/home/fred/dev/sage/local/share/sage/ext -I/home/fred/dev/sage/src/sage/ext -Imtx2.2.4/src -IpGroupCohomology/c_sources -IpGroupCohomology -I/home/fred/dev/sage/local/include/python2.7 -c pGroupCohomology/cohomology.c -o build/temp.linux-x86_64-2.7/pGroupCohomology/cohomology.o -O3
In file included from pGroupCohomology/cohomology.c:247:0:
mtx2.2.4/src/meataxe.h:15:0: warning: "_GNU_SOURCE" redefined
 #define _GNU_SOURCE
 ^
In file included from pGroupCohomology/cohomology.c:8:0:
/home/fred/dev/sage/local/include/python2.7/pyconfig.h:1160:0: note: this is the location of the previous definition
 #define _GNU_SOURCE 1
 ^
pGroupCohomology/cohomology.c:262:37: fatal error: sage/libs/ntl/ntlwrap.cpp: No such file or directory
 #include "sage/libs/ntl/ntlwrap.cpp"
                                     ^
compilation terminated.
error: command 'gcc' failed with exit status 1
Error installing Cython code.

real    0m43.412s
user    0m40.636s
sys 0m2.736s
************************************************************************
Error installing package p_group_cohomology-2.1.5
************************************************************************

jdemeyer commented 9 years ago

comment:17

You might want to wait for #18494, which should make it easier to install external packages using the Sage library.

simon-king-jena commented 9 years ago

comment:18

The real questions are: Why is it looking for ntl? I don't think it is explicitly requested by my package, thus, it probably is an indirect dependency, i.e., one that is in fact a dependency of something in Sage that I cimport into my package. But if it is a dependency of something in Sage, then why does it not work?

I suppose that I have to provide an include path to fix it. If the dependency really is via something in the Sage library, then this "have to" is not very nice---Sage could figure it out without human help.

jdemeyer commented 9 years ago

comment:19

Replying to @simon-king-jena:

The real questions are: Why is it looking for ntl?

Because the Sage <-> NTL interface is a real mess. In particular, any module which needs just a fragment of NTL automatically cimports essentially all of the NTL interface. In particular, sage.rings.integer does that.

jdemeyer commented 9 years ago

comment:20

In setup.py, replace

              include_dirs = CSAGE_PATH + [
                              os.path.join(SAGE_SRC,"c_lib","include"),
                              SAGE_EXTCODE, SRC_EXT,
                              os.path.join("mtx2.2.4","src"),
                              os.path.join("pGroupCohomology","c_sources"),
                              "pGroupCohomology"]

by

try:
    # Sage >= 6.8
    from sage.env import sage_include_directories
except ImportError:
    # Sage < 6.8
    def sage_include_directories():
        return [
            os.path.join(SAGE_LOCAL, "include", "csage"),
            os.path.join(SAGE_SRC, "c_lib", "include"),
            SAGE_EXTCODE, SRC_EXT]  # Not sure that you really need SAGE_EXTCODE

...

              include_dirs = sage_include_directories() + [
                             os.path.join("mtx2.2.4","src"),
                             os.path.join("pGroupCohomology","c_sources"),
                             "pGroupCohomology"]

and use #18494.

simon-king-jena commented 9 years ago

comment:21

I hope it is safe to make #18494 a dependency. If I understand correctly, there is only a very trivial doctest failure, and so hopefully it will be in Sage soonish.

Meanwhile I almost gave up to make my package backwards compatible with sage-3.x...

simon-king-jena commented 9 years ago

Dependencies: #18494

simon-king-jena commented 9 years ago

comment:22

With sage_include_directories() I will not need to construct CSAGE_PATH, right?

simon-king-jena commented 9 years ago

comment:23

I have updated the spkg, and successfully installed it on the Sage version that is provided by #18494.

jdemeyer commented 9 years ago

comment:24

Replying to @simon-king-jena:

Which brings up the question again why there are all these incompatible changes in Sage.

Essentially, because of #17854 and its sub-tickets. Removing the low-level c_lib (a.k.a. libcsage) did indeed have consequences for all packages compiling against Sage.

They didn't use to occur so frequently in earlier versions of Sage, IIRC.

And they probably will not occur anymore in the future, now that #17854 is dealt with. And if there are changes, now we can just change the function sage_include_directories() from #18494.

edd8e884-f507-429a-b577-5d554626c0fe commented 9 years ago

comment:25

Is there a reason not to make a new-style packages for p_group_cohomology ?

simon-king-jena commented 9 years ago

comment:26

Replying to @sagetrac-tmonteil:

Is there a reason not to make a new-style packages for p_group_cohomology ?

Yes. I need someone who explains to me in ALL detail how to create a new-style package.

It begins with: How do I create an "upstream" repository for my sources? Note that I will not change the meataxe matrix wrapper that is included in the package into a wrapper for the latest meataxe upstream, as this would require to rewrite most of the c- and much of the cython-code. So, the files in the current package are an old version of meataxe with massive patches (i.e., basically a fork), plus c- and cython- and singular- and gap-code, for which this package currently is the only source.

tscrim commented 9 years ago

comment:27

You create a folder in the $SAGE_ROOT/build/pkgs with files similar to the others. You will also need to put the tarball with the source code in $SAGE_ROOT/upstream (and post it here or online so that it can be put on the online repo once merged) and then run sage -sh sage-fix-pkg-checksums. All of this folder should be under version control.

simon-king-jena commented 8 years ago

comment:28

It seems that there is no way around creating a new-style spkg, although I still find it odd to artificially create an upstream source in a situation where the spkg IS upstream. Sigh.

I have to

get a github account (or reactivate an existing account?),
learn how to convert the existing hg repository in the spkg to git,
look up how to push my private repository to github,
do what is said in comment:27.

Did I forget something?

Now comes the complicated part.

The spkg has three sources, namely

C-MeatAxe
C programs written by David Green
Cython, Python, Gap and Singular code written by me.

Ad 1.

Actually the starting point is a very old version of C-MeatAxe. If I see that correctly, upstream currently isn't very active. However, compared with the version used in the spkg, upstream has changed most function names.

I apply big patches to C-MeatAxe, as I implemented Strassen multiplication. It improves performance a lot (in the old sources). However, Strassen still is not implemented in the current upstream sources. Two years ago I attended a presentation of one C-MeatAxe developer, and he said that they prefer to optimise the existing implementation rather than adding asymptotically fast implementations. It didn't convince him when I told him that in the old MeatAxe, Strassen beats school book multiplication for matrices as small as 200x200.

Ad 2.

David Green's programs have never been published independently. Hence, the spkg is upstream. The code uses the old C-MeatAxe extensively.

Ad 3.

My code has never been published independently. Hence, the spkg is upstream. It contains a Cython wrapper for the old C-MeatAxe.

Conclusion

I guess in an ideal world, I would create a new spkg for C-MeatAxe, containing my patches and my Cython wrapper and the MeatAxe executable, but starting with current upstream. I would then create a repository for Green's and my code. p_group_cohomology would then depend on the new MeatAxe spkg.

If I recall correctly, I actually have a wrapper for a recent version of MeatAxe, perhaps even with my Strassen implementation. Hence, creating the new MeatAxe spkg might be a feasible first step.

However, rewriting Green's programs in terms of the new MeatAxe could be a pain in the neck.

So, as a short term solution, I'd like to keep the proposed version 2.1.5 of my spkg, that still uses the old C-MeatAxe, with the prospect that version 3.0 should be based on the current MeatAxe upstream.

simon-king-jena commented 8 years ago

comment:29

Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.

tscrim commented 8 years ago

comment:30

I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).

simon-king-jena commented 8 years ago

Description changed:

--- 
+++ 
@@ -2,4 +2,4 @@

 Hence, an upgrade is needed. While I was at it, I have also improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

-The new spkg is at http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.5.spkg
+The new spkg is at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg

dimpase commented 8 years ago

comment:32

Replying to @simon-king-jena:

Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.

how about you make a copy of your spkg on geom.math.washington.edu or combinat.math.washington.edu somewhere readable, e.g. to /tmp ?

At least then we can pick it up and put up somewhere...

simon-king-jena commented 8 years ago

comment:33

Replying to @tscrim:

I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).

Thank you! For now, I have put the old-style spkg at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg

simon-king-jena commented 8 years ago

comment:34

PS: The spkg uses a data base of pre-computed cohomology rings that used to be at sage.math.washington. Hence, now it is not accessible.

If the spkg can't find a cohomology ring in the data base, then it will compute it from scratch. So, most self-tests should work. However, some tests explicitly try to find data in the sage.math.washington data base. These will fail.

It seems that the data base (20-30GB) can be relocated to Jena as well. But it will take at least till next week, and when it's done I need to change the spkg to using a new default url of the data base.

simon-king-jena commented 8 years ago

comment:35

From the README file of the latest upstream sources (which is version 2.4.24 from October 2011(!)).

================================================================================
INSTALLATION (for Unix systems)
================================================================================

1) Copy Makefile.conf.dist to Makefile.conf. Edit Makefile.conf
   and fill in the necessary parameters.

Is that how it is supposed to work with a modern build system?

simon-king-jena commented 8 years ago

comment:36

There is #12103 for using MeatAxe implementation of matrix multiplication over small finite non-prime fields. But that was based on version 2.2.4, not 2.4.xx.

jhpalmieri commented 8 years ago

comment:37

Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology with spkg-install etc., and (2) a tarball containing the upstream source. Since you are the maintainer, it is up to you how to distribute that tarball, so I think if you want to distribute it with an hg repository, that's okay. If you just want to release a particular version with no repo at all included, that should be fine, too. (We don't always distribute repository information in the tarballs for other packages, for what that's worth.)

jhpalmieri commented 8 years ago

comment:38

For now, by the way, I think the easiest path is the right one: use your current versions of everything, don't bother trying to use newer versions of MeatAxe, etc.

simon-king-jena commented 8 years ago

comment:39

Replying to @jhpalmieri:

Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology with spkg-install etc., and (2) a tarball containing the upstream source.

Is it that easy? That would be nice.

I was told somewhere that moving from hg (even with queues?) to git is not particularly difficult. I was reluctant to use git for Sage development and I still miss hg sometimes. But using git for Sage and at the same time hg for a Sage package seems awkward to me

tscrim commented 8 years ago

comment:40

I agree with John: try to keep things as intact as possible. It should be possible to simply detach the SPKG.txt and sage-install scripts from spkg and put those (and sage-check and any patches) in build/pkgs/p_group_cohomology (with a package-version.txt, checksums.ini, and type files; see any other spkg). Then run sage -sh sage-fix-pkg-checksums. You should put a clean tarball of the necessary source in the upstream folder (this will be the upstream tarball that will go on the Sage mirrors), which should be everything else. Then, and only then, check that it installs (otherwise I found it deletes the tarball).

Just to be clear, I think it would be better to simply transition the spkg style here and now and consider doing upgrades later (with a version bump of your spkg).

Also have you considered bitbucket? They do hg repositories and you can keep your upstream there (and under hg control instead of git). Also they seem to be more generous than github for free (private) academic repositories.

jdemeyer commented 8 years ago

comment:41

So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package? In the latter case, the ticket should be set back to needs_work.

simon-king-jena commented 8 years ago

comment:42

Replying to @jdemeyer:

So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package?

If I understood correctly, the suggestion of John and Travis was to create a new-style package with the least effort. I.e., take the current sources contained in the spkg and post them somewhere (aka "upstream"), and install new-stylish from there.

Or perhaps a little better and still feasible: Use the proper existing upstream for MeatAxe (even though the package uses a very old version), and only put the rest of the sources to a new upstream.

In the latter case, the ticket should be set back to needs_work.

OK.

simon-king-jena commented 8 years ago

Work Issues: Create a new-style package with least effort

simon-king-jena commented 8 years ago

comment:43

I have moved the documentation of the package. In case you are interested in testing an old-style version 2.1.5, see http://users.minet.uni-jena.de/cohomology/documentation/#installation

Next, I plan to create a least-effort new-style version 2.1.6. It will be basically identical with 2.1.5, except for the package style. And changing everything to the latest MeatAxe upstream will be considerable work taking some time, justifying a new ticket and a major new version 3.0.

simon-king-jena commented 8 years ago

comment:44

Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.

simon-king-jena commented 8 years ago

comment:45

Sigh. Apparently there is also a change in the pickling of Singular objects that I am responsible for and that I have now to cope with...

jdemeyer commented 8 years ago

comment:46

Replying to @simon-king-jena:

Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.

I guess it's because of the Cython upgrade. Earlier versions of Cython did not give an error for code like

f(foo="hello", foo="world")

but now that's an error just like in Python.

simon-king-jena commented 8 years ago

comment:47

Replying to @jdemeyer:

I guess it's because of the Cython upgrade.

Probably. That error (somewhere doing mutable = True, mutable = True) is fixed. But the Singular pickling error is likely to be tricky.

Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.

sagemath / sage

Upgrade of p_group_cohomology spkg #18514