Upgrade of p_group_cohomology spkg

[simon-king-jena]

In the past, p_group_cohomology-2.1.4 was an optional spkg. Sage dropped support for old style spkgs, and thus only an experimental spkg remained.

The purpose of this ticket is to provide a new group cohomology package for Sage. I suggest to make it optional (not experimental).

In the experimental version 2.1.5, I have improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

Package structure

Version 3.0 is refactored as follows:

• It depends on the optional meataxe package (see #24359), which is based on a fork of MeatAxe.
• The spkg-install script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
• The C code written by David Green has become an autotoolized shared library.
• Gap and Singular code is put into SAGE_SHARE/, where GAP and Singular libraries belong.
• The rest of the package is formed by python and cython code, and uses pip for installation.
• The documentation is built in an improved way.

Further changes

• The custom logging in the old spkg is replaced by Python's logging module.
• The tests avoid side-effects by using a reset function. Hence, no custom doc tester is needed: sage -t is enough.
• In the previous spkg, I tried to be as backward compatible as possible. But now I'm dropping support for very old Singular versions.
• Some experimental options for the cohomology computation are removed.

Many tests have changed, since the ring presentations of the computed cohomology rings, specifically for non-primepower groups, have changed. I do not fully understand why they changed. but it seems that it is a consequence of changes in Singular. The changes did (of course...) not affect the isomorphism types of the rings.

Upstream tar ball

https://users.fmi.uni-jena.de/cohomology/p_group_cohomology-3.0.tar.xz

Upstream: None of the above - read trac for reasoning.

CC: @jhpalmieri @vbraun @jdemeyer @haraldschilly @frederichan-IMJPRG david.green@uni-jena.de

Component: packages: optional

Keywords: group cohomology

Author: Simon King

Branch: 6576915

Reviewer: Travis Scrimshaw, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/18514

[jdemeyer]

comment:48

Replying to @simon-king-jena:

Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.

I'm afraid I can't help you there...

[simon-king-jena]

comment:49

In the previous versions, I already had a class GapPickler to help with pickles of Gap interface objects (by default, their content is lost when pickling, although it can be reconstructed from the string representation in most cases). I guess I could do basically the same for other interfaces.

[simon-king-jena]

comment:50

I replaced the spkg by a version that almost passes the test suite. The only remaining problem is that in the stored data the old pickling of Singular elements is used, which results in a deprecation warning.

So, my plan is: Re-compute the cohomology for all groups of order 64, store them in a database, put the database into the package, and put it online.

Then, finally create a basic version of a new-style package.

[simon-king-jena]

comment:51

I created the current version 2.1.5 on my laptop and tested on the computer in my office. Most tests passed. Problems:

• It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.

• There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before re-installing the package, then the tests will work.

• There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.

[simon-king-jena]

comment:52

Replying to @simon-king-jena:

• It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.

Strangely, these tests worked on my laptop. Is it possible that "screen" (which I use on the office computer) prevents internet access during doctests?

• There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before re-installing the package, then the tests will work.

Question to the reviewer: Should one add a remark in the documentation, telling that a certain deprecation warning is expected if old data is present?

• There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.

I get the same on my laptop. Thus I changed it, and updated the package. Now I redo the tests on my office computer.

[tscrim]

comment:53

Is there anything I can do to help right now?

[simon-king-jena]

comment:54

I guess I have to do some more fixes. It turns out that one can not install the spkg if before one has installed the meataxe package that I propose at #12103. Probable reason: Conflicting types in the meataxe version used here and used there.

Actually the cleanest solution would be to remove the old meataxe sources from the spkg, and rewrite the code of David Green by using the new meataxe sources. Hence, the meataxe package would become a dependency of the cohomology spkg.

But my plan was to do this (a MAJOR rewrite!) on the long run, not now. I will do some tests, to see how much work will be involved. If it merely is a cut-and-paste modification (changing ALL function and type names used in meataxe) then it might be doable.

[tscrim]

comment:55

A more hack solution would be to make the meataxe that you include in this spkg have a different name, and so that it could be used along #12103. However, as you said, the cleanest solution would be to base this off of the meataxe spkg on #12103 as a dependency.

I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...

[simon-king-jena]

comment:56

Replying to @tscrim:

I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...

When translating my implementation of Winograd-Strassen multiplication from meataxe-2.2.4 to meataxe-2.4.24, most of the work has indeed been cut-and-paste.

However, we talk here about changing third party code (namely David Green's), and since I didn't write it I don't know if the transition will be smooth.

[simon-king-jena]

comment:57

At least the data structure has remained basically the same (representing matrices as a contiguous block of memory organised in rows of the matrix, each byte contains up to 8 matrix coefficients, depending on the field size, and all arithmetic operations are done by table lookup).

[simon-king-jena]

comment:58

I am now confident that it is feasible to make the group cohomology software work with the new optional meataxe package from #12103. I managed to rewrite the stand-alone programs of David Green needed in the package, and the rest "should work" (TM) by search-and-replace operations (really, ALL names in meataxe have changed).

But I have questions concerning the new code structure.

The old spkg (p_group_cohomology-2.1.5) consists of

• Original sources of meataxe-2.2.4, together with my "fork" of meataxe,
• C-programs written by David Green, in src/present/,
• GAP-functions written by David Green, in src/pGroupCohomology,
• Singular-functions written by me, in src/pGroupCohomology,
• Cython modules written by me, in src/pGroupCohomology,
• documentation, including a doc builder taken (and modified) from old Sage versions,
• a data base of cohomology rings for the groups of order 64.

For the new package, I suggest:

• meataxe-2.4.24 is used, as provided by the package from #12103.
• The C-programs of David Green, converted to work with the new MeatAxe, will be a new optional package called modular_resolutions-1.0 (or do you prefer modres-1.0?) depending on meataxe, adding me as a second author, as the conversion to the new MeatAxe was very nontrivial, and I also added error handling.
• The database will also be part of modular_resolutions.
• The Cython modules will become optional extensions in the SageMath library, say, sage.groups.modular_cohomology, depending on modular_resolutions and on database_gap. In that way, the documentation would be taken care of.
• Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

Remark: database_gap is needed because of the SmallGroups library. Technically, the upcoming modular_resolutions (or modres) package would not depend on it; but it is needed in the cohomology computations, thus, perhaps it makes still sense to add database_gap as a dependency of modular_resolutions?

[jdemeyer]

comment:59

Replying to @simon-king-jena:

or do you prefer modres-1.0?

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

[jdemeyer]

comment:60

Replying to @simon-king-jena:

• Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use src/ext/gap?

[simon-king-jena]

comment:61

Replying to @jdemeyer:

Replying to @simon-king-jena:

or do you prefer modres-1.0?

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

Concerning "upstream": David Green is the original upstream. But as much as I know his code has never been published outside of the old spkg. And since the rebase on top of meataxe-2.4.24 was major, I believe that I am to some extent upstream, too.

The original sources that David Green gave me are in a folder called "present". But I doubt that a user having a look at the list of optional packages would guess that "present" is about modular resolutions of p-groups. From my perspective, "modular_resolutions" is a bit lengthy. I will ask David, after all he is native speaker.

[simon-king-jena]

comment:62

Replying to @jdemeyer:

Replying to @simon-king-jena:

• Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use src/ext/gap?

Pure GAP code, in the files src/pGroupCohomology/GapMB, .../GapMaxels and .../GapSgs.

Is src/ext/gap somehow known to Gap? Or does one need to give the full path when loading code from there into Gap?

[dimpase]

comment:63

you might consider packaging your GAP code as a GAP package...

[simon-king-jena]

comment:64

Replying to @dimpase:

you might consider packaging your GAP code as a GAP package...

Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.

[simon-king-jena]

comment:65

Replying to @simon-king-jena:

Replying to @jdemeyer:

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

I got answer of David Green.

• He agrees that, by porting the code on top of the new meataxe, I became a co-author.
• He coined his package "present", because the original purpose was to deal with presentations of modular group algebras of prime power groups.
• If I understand correctly, he wouldn't mind renaming it.

I believe that the main purpose of the code is not to compute a presentation of a modular group algebra, but to compute a minimal projective resolution of the modular group algebra. At least that's what I am using it for.

Since the old package is called p_group_cohomology and the cohomology part will be moved to the Sage library, the package's responsibility will be to compute resolutions. Perhaps (to remind the old name) p_resolution? I find modular_resolution too long, Jeroen finds modres too abbreviated (I guess).

[dimpase]

comment:66

Replying to @simon-king-jena:

Replying to @dimpase:

you might consider packaging your GAP code as a GAP package...

Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.

well, by package I just meant a way to package GAP files in a way understood by GAP.

Of course, you can also use GAP's Read() command to load any GAP code from anywhere in the filesystem.

sage: gap.eval('Read("'+SAGE_LOCAL+'/foo/blah.g")') # GAP
sage: libgap.Read('"'+SAGE_LOCAL+'/foo/blah.g"') # libGAP

HTH, Dima

[dimpase]

comment:67

by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?

[simon-king-jena]

comment:68

Replying to @dimpase:

by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?

No problem. And then put it into src/ext/gap, as I was told.

[simon-king-jena]

comment:69

I think I need help, as I am virtually a beginner with Makefiles, compiling c-sources, and/or creating a library.

For creating .o-files, I have the following rule in my Makefile, where

• VPATH is the path to the sources
• CFLAGS=-I$(MTXINC) -I$(VPATH) -Wall $(OO) -g -fPIC
• MTXINC is the SAGE_LOCAL/include

%.o: $(VPATH)/pgroup.h $(VPATH)/pincl.h $(VPATH)/pincl_decls.h $(VPATH)/$*.c
    @echo "Compiling $*.c"
    @$(CC) $(CFLAGS) -c $(VPATH)/$*.c -o $@

The above is a modification of MeatAxe's Makefile, which says

tmp/%.o: tmp/mk.dir src/%.c src/meataxe.h tmp/config.h
    @echo "Compiling $*.c"
    @$(CC) $(CFLAGS) -c src/$*.c -o $@

While the rule in MeatAxe's Makefile works, in my Makefile it is not executed. For example, when making pinc.o, the following is printed to stdout:

gcc -I/home/king/Sage/git/sage/local/include -I/home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src -Wall  -g -fPIC    -c -o pincl.o /home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src/pincl.c

In particular, the @echo "compiling ..." statement is not executed.

So, what is happening here? Is there an implicit make rule that takes precedence over the explicit rule??

And further, when I try to build an executable (say, makeInclusionMatrix), the rule says

makeInclusionMatrix: libmodres.a mim.o
    $(CC) $(LFLAGS) -o makeInclusionMatrix mim.o libmodres.a

with

• LFLAGS=-l$(MTXLIBNAME) -L$(LIBDIR),
• MTXLIBNAME=mtx
• LIBDIR=$(SAGE_LOCAL)/lib Building the executable, the output indeed contains

gcc -lmtx -L/home/king/Sage/git/sage/local/lib  -o makeInclusionMatrix mim.o libmodres.a

but aborts with an error, saying

/home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src/mim.c:51: undefined reference to `MatFree'

But MatFree and other "undefined references" are defined in /home/king/Sage/git/sage/local/lib/libmtx.a!

What is happening here?

[simon-king-jena]

comment:70

When I instead do

gcc -L/home/king/Sage/git/sage/local/lib  -o makeInclusionMatrix mim.o pgroup.o pincl.o -lmtx

then it works. But why? And why did the other command not work?

The difference is that it says pgroup.o pincl.o instead of libmodres.a. And libmodres.a was created with ar r libmodres.a pincl.o pgroup.o aufloesung.o. Shouldn't then pgroup.o and pincl.o be available in libmodres.a anyway?

[jdemeyer]

comment:71

Replying to @simon-king-jena:

I think I need help, as I am virtually a beginner with Makefiles, compiling c-sources, and/or creating a library.

If you're manually writing a Makefile, you're already doing it wrong. The only way to reliably build a library is to use libtool.

[simon-king-jena]

comment:72

Replying to @jdemeyer:

If you're manually writing a Makefile, you're already doing it wrong. The only way to reliably build a library is to use libtool.

OK, then MeatAxe is doing it wrong as well, but after all it is third party code and I won't touch it.

Can you point me to a good introduction to libtool? What I need to create is (a) a couple of executables and (b) a library against which I can link my cython code. So, the library is not all!

[jdemeyer]

comment:73

Replying to @simon-king-jena:

OK, then MeatAxe is doing it wrong as well

Absolutely. Don't use the meataxe build system as an example.

Can you point me to a good introduction to libtool?

I don't know. For an example in Sage, you can look at the planarity package.

[dimpase]

comment:74

Replying to @jdemeyer:

Replying to @simon-king-jena:

OK, then MeatAxe is doing it wrong as well

Absolutely. Don't use the meataxe build system as an example.

Can you point me to a good introduction to libtool?

I don't know. For an example in Sage, you can look at the planarity package.

as far as I can see, meataxe does not depend on any external libraries, right? Then it's really easy to autotoolize/libtoolize. An example of a Sage package I can suggest is csdp (well, I did this libtoolization for it :-)) https://github.com/dimpase/csdp it's more complicated as it needs Lapack and Blas.

I'll email you a pdf of a readable book about autotools, so that you can get an idea what's going on. (there is no readable free online up to day text for these purposes, AFAIK).

[simon-king-jena]

comment:75

Still I'd like to understand what went wrong in the scenario of comment:69 and comment:70. After all, before I can autotoolize it, I should probably first be able to build it.

[vbraun]

comment:76

As to what went wrong, I think gcc will not use static libraries if you just list them on the command line. You need -lmodres.

While building binaries is somewhat the same on all platforms, the process of building a shared library is very platform dependent. Hence Jeroen's very good advice about libtool.

[simon-king-jena]

comment:77

Meanwhile I really see the point of using autotools. Perhaps the experts (i.e. you) have comments/suggestions on the following, but to me it seems to work.

src/Makefile.am

AM_CFLAGS = -O3

# -----> Shared library
lib_LIBRARIES               = libmodres.a
libmodres_a_SOURCES         = pincl.c pgroup.c aufloesung.c

# -----> Headers
include_HEADERS             = modular_resolution.h

# -----> Executable (built from the shared library)

bin_PROGRAMS = makeActionMatrices makeNontips groupInfo perm2Gap makeInclusionMatrix
makeActionMatrices_SOURCES  = mam.c
makeActionMatrices_LDADD    = $(lib_LIBRARIES)
makeNontips_SOURCES         = mnt.c
makeNontips_LDADD           = $(lib_LIBRARIES)
groupInfo_SOURCES           = gi.c
groupInfo_LDADD             = $(lib_LIBRARIES)
perm2Gap_SOURCES            = perm2Gap.c
perm2Gap_LDADD              = $(lib_LIBRARIES)
makeInclusionMatrix_SOURCES = mim.c
makeInclusionMatrix_LDADD   = $(lib_LIBRARIES)

configure.ac

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.69])
AC_INIT([modular_resolution], [1.0], [simon.king@uni-jena.de])
AM_INIT_AUTOMAKE
LT_INIT
AC_CONFIG_SRCDIR([src/fp_decls.h])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC
AC_PROG_INSTALL

# Checks for libraries.
AC_SEARCH_LIBS([MtxError], [mtx])

# Checks for header files.
AC_CHECK_HEADERS([stdlib.h string.h unistd.h meataxe.h])

# Checks for typedefs, structures, and compiler characteristics.
AC_CHECK_HEADER_STDBOOL
AC_C_INLINE
AC_TYPE_SIZE_T

# Checks for library functions.
AC_FUNC_MALLOC

AC_CONFIG_FILES([Makefile src/Makefile])
AC_OUTPUT

So far I only see a single change needed: My email address will very soon change.

[simon-king-jena]

comment:78

Can you help me to solve the following problem, please?

I would like to add a little test script that relies on a file test.reg that can be found in the same directory as modular_resolution.h.

In configure.ac, I have AC_CONFIG_SRCDIR([src/modular_resolution.h])

When I go to a new directory and do

/path/to/sources/of/modular_resolution/configure --prefix=`pwd`

then make check works. The file test.reg is sought in /path/to/source/of/modular_resolution/src, where it can indeed be found.

However, when I do make distcheck (which is a target provided by autotools), then first the distdir is created, and in distdir/_build the package is built and tested. Building it works fine. However, the test fails, because this time test.reg is sought in ../../src. Apparently the test is executed in distdir/_build, not in distdir/_build/src. Thus, it should be sought in ../src.

In other words, when doing "make check", an absolute path is used, whereas in "make distcheck" a wrong relative path is used.

How can that be fixed?

[jdemeyer]

comment:79

Use $(srcdir) to refer to the source directory in Makefile.am

Another thing: avoid AM_CFLAGS = -O3. Just pass whatever flags you want for Sage to configure.

[simon-king-jena]

comment:80

Replying to @jdemeyer:

Use $(srcdir) to refer to the source directory in Makefile.am

I did use it. Meanwhile I could fix the problem: the test script was in fact passing a wrong command line argument to some executable.

[simon-king-jena]

comment:81

How to indicate that the package-to-be depends both on meataxe and I guess as well on some toolchain? I see the following in some dependencies files:

$(INST)/$(SAGE_MP_LIBRARY) $(INST)/$(ZLIB)

----------
All lines of this file are ignored except the first.
It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.

or

$(INST)/$(MARKUPSAFE) $(INST)/$(SETUPTOOLS) $(INST)/$(DOCUTILS)

----------
All lines of this file are ignored except the first.
It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.

The developer manual doesn't seem to mention it.

[simon-king-jena]

comment:82

Meanwhile I have a version of the C-part of the old group cohomology package that installs fine and even passes a self-test (namely tests that several of the installed executables work together to build the basic data for the cohomology computation of the dihedral group of order 8).

However, so far I have no idea how to make it so that the package is only built if meataxe (from #12103) is installed.

And then of course I have to put all Singular and Gap files with the appropriate file name extensions into the appropriate folders, and put the Cython and Python code into the Sage library, as an optional extension.

If I understand correctly, it is needed to add "optional: modular_resolution" basically to each single line of test (several hundreds, I guess). Is there really no way to state that a whole file should only be tested optionally? I believe it is a fairly natural thing to test an optional extension optionally.

[simon-king-jena]

Branch: u/SimonKing/upgrade_of_group_cohomology_spkg

[simon-king-jena]

comment:84

I have extracted the second stand-alone ingredient of the group cohomology package.

• First, install meataxe (see #12103).
• Second, get the branch from here.
• Third, get the sources for modular_resolution-1.0. It is based on the original source code of David Green, which I refactored rather extensively: It now uses the optional meataxe-2.4.24 package, rather than meataxe-2.2.4 sources that used to be part of the old spkg. It propagates errors (at least to some extent). And it is autotoolized. I am upstream, and you find the tarball at http://users.minet.uni-jena.de/cohomology/ (To be precise: http://users.minet.uni-jena.de/cohomology/modular_resolution-1.0.tar.bz2).

The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.

And the final step will be to fix the doctests somehow. I think there should be a way to declare that the tests of all files in sage.group.modular_cohomology are optional.

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages? Or better move the dependency to the wrapper?

---

Last 10 new commits:

c7d75fb	Implement and use Strassen-Winograd matrix multiplication in MeatAxe
9e2a8c6	A very basic MeatAxe Cython wrapper
4bdd285	A full wrapper for MeatAxe matrices
d889e0b	Improve echelon computation in MeatAxe, and fix some compiler warnings
2e64257	Doctests and error handling for MeatAxe
7c38969	Fix computation of row-reduced echelon form
55a278d	Fix doctests when meataxe is installed
f733377	Use and propagate specific return values on error in matrix-related MeatAxe functions.
efa1d75	Merge branch 't/12103/meataxe' into t/18514/upgrade_of_group_cohomology_spkg
ad6cea0	Add package modular_resolution

[simon-king-jena]

Commit: ad6cea0

[jdemeyer]

Description changed:

--- 
+++ 
@@ -1,3 +1,5 @@
+**OUTDATED:**
+
 The current "official" version of `p_group_cohomology` is 2.1.4. However, due to recent backward incompatible changes in Sage, the package would not install, respectively it would install if some header were present but wouldn't work.

 Hence, an upgrade is needed. While I was at it, I have also improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

[jdemeyer]

comment:85

Please update the ticket description (the last comment you made is a good starting point).

[jdemeyer]

comment:86

Replying to @simon-king-jena:

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?

No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.

Or better move the dependency to the wrapper?

The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?

[simon-king-jena]

comment:87

Replying to @jdemeyer:

Replying to @simon-king-jena:

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?

No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.

Then we might be in trouble. If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Perhaps I should elaborate on the dependency on database_gap. The code does not link against database gap. However, it needs to be able to deal with elementary abelian groups. Sounds trivial, and it can certainly be solved abstractly, but for now the elementary abelian group is simply looked up in the small groups library.

Also, since the p_group_cohomology package is some kind of data base for cohomology rings, it is needed to assign a unique identifier to a group-with-a-fixed-minimal-generating-set. That is easier with the small groups library, but the package also allows to assign a name to a group (e.g., Co_3 for the third Conway group), and use the name as unique identifier for some item in the cohomology ring database (the fixed minimal generating set is stored along with the ring).

Or better move the dependency to the wrapper?

The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?

Similarly to the MeatAxe wrapper, I plan to use OptionalExtension(). Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.

So, if the current framework does not allow that installation of modular_resolutions stops for asking the users permission to install database_gap first, then we can make it so that modular_resolution only makes sure that meataxe is installed (no problem, both are GPL compatible, and modular_resolution will build and test fine), and then the OptionalExtension() will depend on both modular_resolution and database_gap.

[jdemeyer]

comment:88

Replying to @simon-king-jena:

If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Where did you get this idea from?

[simon-king-jena]

Description changed:

--- 
+++ 
@@ -1,7 +1,21 @@
-**OUTDATED:**
-
-The current "official" version of `p_group_cohomology` is 2.1.4. However, due to recent backward incompatible changes in Sage, the package would not install, respectively it would install if some header were present but wouldn't work.
+The current "official" version of `p_group_cohomology` is 2.1.4. However, due to recent backward incompatible changes in Sage, the package would not install, respectively it would install if some header were present but wouldn't work. Note that such backward incompatible changes happened repeatedly.

 Hence, an upgrade is needed. While I was at it, I have also improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

-The new spkg is at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg
+There is an old-style spkg at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg that users can install if they want to do cohomology calculations *now*. However, I suggest to a version 3.0 of the package, which shall be a new-style package.
+
+The new package shall be modularised as follows.
+
+- First, install meataxe (see #12103).
+- Second, get the branch from here.
+- Third, get the sources for modular_resolution-1.0. It is based on code of David Green, which I refactored rather extensively:
+  * It now uses the optional meataxe-2.4.24 package, rather than meataxe-2.2.4 whose sources used to be part of the old spkg.
+  * It propagates errors (at least to some extent). The original version would just crash if a file was missing.
+  * It is autotoolized.
+
+I am upstream, and you find the tarball at http://users.minet.uni-jena.de/cohomology/ (To be precise: http://users.minet.uni-jena.de/cohomology/modular_resolution-1.0.tar.bz2).
+
+The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.
+
+And the final step will be to fix the doctests somehow. I think there should be a way to declare that the tests of all files in sage.group.modular_cohomology are optional.
+

[simon-king-jena]

comment:90

Replying to @jdemeyer:

Replying to @simon-king-jena:

If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Where did you get this idea from?

From discussions on sage-devel.

[simon-king-jena]

Description changed:

--- 
+++ 
@@ -6,7 +6,7 @@

 The new package shall be modularised as follows.

-- First, install meataxe (see #12103).
+- First, install meataxe (see #12103) and database_gap.
 - Second, get the branch from here.
 - Third, get the sources for modular_resolution-1.0. It is based on code of David Green, which I refactored rather extensively:
   * It now uses the optional meataxe-2.4.24 package, rather than meataxe-2.2.4 whose sources used to be part of the old spkg.

[jdemeyer]

comment:92

Replying to @simon-king-jena:

From discussions on sage-devel.

I recall that discussion and I asked exactly the same question. I don't think that was ever answered. I think it was even mentioned that we should only ask this question as a courtesy to the user, not that it was a requirement.

[jdemeyer]

comment:93

Replying to @simon-king-jena:

Similarly to the MeatAxe wrapper, I plan to use OptionalExtension(). Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.

Surely, the module will compile without database_gap. So I think you should just add meataxe as dependency for the Cython module.

The need of database_gap will make it fail at runtime. That's no problem. It's even more user-friendly, since you can show a nice error message at runtime if database_gap is not installed.

[simon-king-jena]

comment:94

Replying to @jdemeyer:

The need of database_gap will make it fail at runtime. That's no problem. It's even more user-friendly, since you can show a nice error message at runtime if database_gap is not installed.

OK, sounds good, except for doctests: It means that all tests in the OptionalExtension need to be marked as optional: modular_resolution database_gap, which is rather awkward to do hundreds of times. Unless of course one can somehow declare that ALL tests of a file are optional.

[jdemeyer]

comment:95

Replying to @simon-king-jena:

It means that all tests in the OptionalExtension need to be marked as optional: modular_resolution database_gap, which is rather awkward to do hundreds of times.

Well, it's not more awkward than just optional: modular_resolution hundreds of times :-)

Unless of course one can somehow declare that ALL tests of a file are optional.

I think this is intentionally not supported. A doctest should make sense by itself, not depend on external context like the file it appears in.