Closed simon-king-jena closed 6 years ago
For me, all but two tests fail. Both tests fail with "connection timed out" during access to a web data base, and both work fine when done in an interactive session.
Has something in the test framework changed, so that web access via urllib2
fails during tests?
I see you are compiling some extensions with the csage
library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage
only if it actually exists.
Replying to @jdemeyer:
I see you are compiling some extensions with the
csage
library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile againstcsage
only if it actually exists.
OK. So I should see what I actually use of csage.
The sig_on()
framework was in csage
until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage
if the Sage version is <= 6.7
Replying to @jdemeyer:
The
sig_on()
framework was incsage
until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link againstcsage
if the Sage version is <= 6.7
OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off
, or would this just work automatically with the new Cython version?
Replying to @simon-king-jena:
OK. What does that mean for the code? Do I need to import anything if I am using
sig_on/off
, or would this just work automatically with the new Cython version?
You just include sage/ext/interrupt.pxi
, that doesn't change.
The main difference is that you no longer need to add csage
as library. Currently, it doesn't hurt if you add the library anyway, but in the future, csage
might not exist anymore.
And there is one additional caveat which is relevant for your package: since #18027, you must include interrupt.pxi
only in .pyx
files and not .pxd
files (unless you really know what you're doing).
On the topic of .pxd
files, this is what they should contain:
The API of your module, that is all cdef
classes and c(p)def
functions that you want to export for use in other modules (in the same package or by other packages).
If your module provides bindings to some library (the mtx
module in your package for example), then add the declarations of the library functions in the .pxd
file.
Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from RingElement
. In your .pxd
file, you want to write cdef class MySpecialElement(RingElement)
. This makes RingElement
a dependency, so you need to write from ... cimport RingElement
in the .pxd
file.
And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx
file. This is important in order to minimize dependencies.
NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd
file. In Sage, we usually put the library bindings in sage/libs
, look at sage/libs/gmp
for a good example.
Replying to @jdemeyer:
On the topic of
.pxd
files, this is what they should contain:
The API of your module, that is all
cdef
classes andc(p)def
functions that you want to export for use in other modules (in the same package or by other packages).If your module provides bindings to some library (the
mtx
module in your package for example), then add the declarations of the library functions in the.pxd
file.Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from
RingElement
. In your.pxd
file, you want to writecdef class MySpecialElement(RingElement)
. This makesRingElement
a dependency, so you need to writefrom ... cimport RingElement
in the.pxd
file.And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the
.pyx
file. This is important in order to minimize dependencies.
OK. That's something that I should change. Actually, until not so long ago, I thought that cimports were only possible in pxd files.
NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different
.pxd
file. In Sage, we usually put the library bindings insage/libs
, look atsage/libs/gmp
for a good example.
Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe
bindings, say, mtx_lib.pxd
, which is then cimported in mtx.pxd
?
Replying to @simon-king-jena:
Do I understand correctly: You suggest that I should create a new pxd file for the
MeatAxe
bindings, say,mtx_lib.pxd
, which is then cimported inmtx.pxd
?
I cannot really speak about this specific case, but it certainly would make sense to split it up.
Work Issues: Cope with yet another backward incompatible change of Sage
I just found that the package won't build with the latest beta of Sage. Reason:
pGroupCohomology/cohomology.c:248:36: fatal error: sage/ext/interrupt/pxi.h: No such file or directory
#include "sage/ext/interrupt/pxi.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
So, can you please tell me how to use interrupts, after all these backward incompatible changes? It really sucks.
Good news! On my machine, I did some refactoring of code according to the hints you gave me---and that did the trick.
My plan is to continue refactoring, taking into account what you said about where to import stuff, and then push a new package version to the official page.
Bad news. It builds, but I can't import it. Which brings up the question again why there are all these incompatible changes in Sage. They didn't use to occur so frequently in earlier versions of Sage, IIRC.
I got it fixed. So, you'll find the updated version of p_group_cohomology-2.1.5
on my web page.
Some of your concerns about importing/including stuff are addressed in the new version: I moved some of it from .pxd files into .pyx files. What I did not do is to split off library bindings and put them into separate .pxd files. That would still make sense, but for now I put the ticket back to "needs review".
Changed work issues from Cope with yet another backward incompatible change of Sage to none
Oh, and I also created a new module pGroupCohomology.auxiliaries
, in which I've put some auxiliary functions that have previously been in pGroupCohomology.resolution
.
I have a built failure for
sage -i upstream/p_group_cohomology-2.1.5.spkg
with sage 6.8beta3+trac18666 (but success with master)
gcc -pthread -shared -L/home/fred/dev/sage/local/lib build/temp.linux-x86_64-2.7/pGroupCohomology/resolution.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufloesung.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufnahme.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/djgerr.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/fileplus.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/nBuchberger.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pgroup.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pincl.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/slice.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/urbild.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/uvr.o -L/home/fred/dev/sage/local/lib -lmtx -lpython2.7 -o build/lib.linux-x86_64-2.7/pGroupCohomology/resolution.so
cythoning pGroupCohomology/cohomology.pyx to pGroupCohomology/cohomology.c
building 'pGroupCohomology.cohomology' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/fred/dev/sage/src/c_lib/include -I/home/fred/dev/sage/local/share/sage/ext -I/home/fred/dev/sage/src/sage/ext -Imtx2.2.4/src -IpGroupCohomology/c_sources -IpGroupCohomology -I/home/fred/dev/sage/local/include/python2.7 -c pGroupCohomology/cohomology.c -o build/temp.linux-x86_64-2.7/pGroupCohomology/cohomology.o -O3
In file included from pGroupCohomology/cohomology.c:247:0:
mtx2.2.4/src/meataxe.h:15:0: warning: "_GNU_SOURCE" redefined
#define _GNU_SOURCE
^
In file included from pGroupCohomology/cohomology.c:8:0:
/home/fred/dev/sage/local/include/python2.7/pyconfig.h:1160:0: note: this is the location of the previous definition
#define _GNU_SOURCE 1
^
pGroupCohomology/cohomology.c:262:37: fatal error: sage/libs/ntl/ntlwrap.cpp: No such file or directory
#include "sage/libs/ntl/ntlwrap.cpp"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
Error installing Cython code.
real 0m43.412s
user 0m40.636s
sys 0m2.736s
************************************************************************
Error installing package p_group_cohomology-2.1.5
************************************************************************
You might want to wait for #18494, which should make it easier to install external packages using the Sage library.
The real questions are: Why is it looking for ntl? I don't think it is explicitly requested by my package, thus, it probably is an indirect dependency, i.e., one that is in fact a dependency of something in Sage that I cimport into my package. But if it is a dependency of something in Sage, then why does it not work?
I suppose that I have to provide an include path to fix it. If the dependency really is via something in the Sage library, then this "have to" is not very nice---Sage could figure it out without human help.
Replying to @simon-king-jena:
The real questions are: Why is it looking for ntl?
Because the Sage <-> NTL interface is a real mess. In particular, any module which needs just a fragment of NTL automatically cimports essentially all of the NTL interface. In particular, sage.rings.integer
does that.
In setup.py
, replace
include_dirs = CSAGE_PATH + [
os.path.join(SAGE_SRC,"c_lib","include"),
SAGE_EXTCODE, SRC_EXT,
os.path.join("mtx2.2.4","src"),
os.path.join("pGroupCohomology","c_sources"),
"pGroupCohomology"]
by
try:
# Sage >= 6.8
from sage.env import sage_include_directories
except ImportError:
# Sage < 6.8
def sage_include_directories():
return [
os.path.join(SAGE_LOCAL, "include", "csage"),
os.path.join(SAGE_SRC, "c_lib", "include"),
SAGE_EXTCODE, SRC_EXT] # Not sure that you really need SAGE_EXTCODE
...
include_dirs = sage_include_directories() + [
os.path.join("mtx2.2.4","src"),
os.path.join("pGroupCohomology","c_sources"),
"pGroupCohomology"]
and use #18494.
I hope it is safe to make #18494 a dependency. If I understand correctly, there is only a very trivial doctest failure, and so hopefully it will be in Sage soonish.
Meanwhile I almost gave up to make my package backwards compatible with sage-3.x...
Dependencies: #18494
With sage_include_directories()
I will not need to construct CSAGE_PATH
, right?
I have updated the spkg, and successfully installed it on the Sage version that is provided by #18494.
Replying to @simon-king-jena:
Which brings up the question again why there are all these incompatible changes in Sage.
Essentially, because of #17854 and its sub-tickets. Removing the low-level c_lib
(a.k.a. libcsage
) did indeed have consequences for all packages compiling against Sage.
They didn't use to occur so frequently in earlier versions of Sage, IIRC.
And they probably will not occur anymore in the future, now that #17854 is dealt with. And if there are changes, now we can just change the function sage_include_directories()
from #18494.
Is there a reason not to make a new-style packages for p_group_cohomology
?
Replying to @sagetrac-tmonteil:
Is there a reason not to make a new-style packages for
p_group_cohomology
?
Yes. I need someone who explains to me in ALL detail how to create a new-style package.
It begins with: How do I create an "upstream" repository for my sources? Note that I will not change the meataxe matrix wrapper that is included in the package into a wrapper for the latest meataxe upstream, as this would require to rewrite most of the c- and much of the cython-code. So, the files in the current package are an old version of meataxe with massive patches (i.e., basically a fork), plus c- and cython- and singular- and gap-code, for which this package currently is the only source.
You create a folder in the $SAGE_ROOT/build/pkgs
with files similar to the others. You will also need to put the tarball with the source code in $SAGE_ROOT/upstream
(and post it here or online so that it can be put on the online repo once merged) and then run sage -sh sage-fix-pkg-checksums
. All of this folder should be under version control.
See also: http://doc.sagemath.org/html/en/developer/packaging.html
It seems that there is no way around creating a new-style spkg, although I still find it odd to artificially create an upstream source in a situation where the spkg IS upstream. Sigh.
I have to
Did I forget something?
Now comes the complicated part.
The spkg has three sources, namely
Ad 1.
Actually the starting point is a very old version of C-MeatAxe. If I see that correctly, upstream currently isn't very active. However, compared with the version used in the spkg, upstream has changed most function names.
I apply big patches to C-MeatAxe, as I implemented Strassen multiplication. It improves performance a lot (in the old sources). However, Strassen still is not implemented in the current upstream sources. Two years ago I attended a presentation of one C-MeatAxe developer, and he said that they prefer to optimise the existing implementation rather than adding asymptotically fast implementations. It didn't convince him when I told him that in the old MeatAxe, Strassen beats school book multiplication for matrices as small as 200x200.
Ad 2.
David Green's programs have never been published independently. Hence, the spkg is upstream. The code uses the old C-MeatAxe extensively.
Ad 3.
My code has never been published independently. Hence, the spkg is upstream. It contains a Cython wrapper for the old C-MeatAxe.
Conclusion
I guess in an ideal world, I would create a new spkg for C-MeatAxe, containing my patches and my Cython wrapper and the MeatAxe executable, but starting with current upstream. I would then create a repository for Green's and my code. p_group_cohomology would then depend on the new MeatAxe spkg.
If I recall correctly, I actually have a wrapper for a recent version of MeatAxe, perhaps even with my Strassen implementation. Hence, creating the new MeatAxe spkg might be a feasible first step.
However, rewriting Green's programs in terms of the new MeatAxe could be a pain in the neck.
So, as a short term solution, I'd like to keep the proposed version 2.1.5 of my spkg, that still uses the old C-MeatAxe, with the prospect that version 3.0 should be based on the current MeatAxe upstream.
Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.
I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).
Description changed:
---
+++
@@ -2,4 +2,4 @@
Hence, an upgrade is needed. While I was at it, I have also improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.
-The new spkg is at http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.5.spkg
+The new spkg is at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg
Replying to @simon-king-jena:
Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.
how about you make a copy of your spkg on geom.math.washington.edu or combinat.math.washington.edu somewhere readable, e.g. to /tmp ?
At least then we can pick it up and put up somewhere...
Replying to @tscrim:
I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).
Thank you! For now, I have put the old-style spkg at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg
PS: The spkg uses a data base of pre-computed cohomology rings that used to be at sage.math.washington. Hence, now it is not accessible.
If the spkg can't find a cohomology ring in the data base, then it will compute it from scratch. So, most self-tests should work. However, some tests explicitly try to find data in the sage.math.washington data base. These will fail.
It seems that the data base (20-30GB) can be relocated to Jena as well. But it will take at least till next week, and when it's done I need to change the spkg to using a new default url of the data base.
From the README file of the latest upstream sources (which is version 2.4.24 from October 2011(!)).
================================================================================
INSTALLATION (for Unix systems)
================================================================================
1) Copy Makefile.conf.dist to Makefile.conf. Edit Makefile.conf
and fill in the necessary parameters.
Is that how it is supposed to work with a modern build system?
There is #12103 for using MeatAxe implementation of matrix multiplication over small finite non-prime fields. But that was based on version 2.2.4, not 2.4.xx.
Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology
with spkg-install
etc., and (2) a tarball containing the upstream source. Since you are the maintainer, it is up to you how to distribute that tarball, so I think if you want to distribute it with an hg repository, that's okay. If you just want to release a particular version with no repo at all included, that should be fine, too. (We don't always distribute repository information in the tarballs for other packages, for what that's worth.)
For now, by the way, I think the easiest path is the right one: use your current versions of everything, don't bother trying to use newer versions of MeatAxe, etc.
Replying to @jhpalmieri:
Do you have to convert the hg repository to a git repository? What we really need is (1) the directory
build/pkgs/p_group_cohomology
withspkg-install
etc., and (2) a tarball containing the upstream source.
Is it that easy? That would be nice.
I was told somewhere that moving from hg (even with queues?) to git is not particularly difficult. I was reluctant to use git for Sage development and I still miss hg sometimes. But using git for Sage and at the same time hg for a Sage package seems awkward to me
I agree with John: try to keep things as intact as possible. It should be possible to simply detach the SPKG.txt
and sage-install
scripts from spkg and put those (and sage-check
and any patches) in build/pkgs/p_group_cohomology
(with a package-version.txt
, checksums.ini
, and type
files; see any other spkg). Then run sage -sh sage-fix-pkg-checksums
. You should put a clean tarball of the necessary source in the upstream
folder (this will be the upstream tarball that will go on the Sage mirrors), which should be everything else. Then, and only then, check that it installs (otherwise I found it deletes the tarball).
Just to be clear, I think it would be better to simply transition the spkg style here and now and consider doing upgrades later (with a version bump of your spkg).
Also have you considered bitbucket? They do hg repositories and you can keep your upstream there (and under hg control instead of git). Also they seem to be more generous than github for free (private) academic repositories.
So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package? In the latter case, the ticket should be set back to needs_work.
Replying to @jdemeyer:
So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package?
If I understood correctly, the suggestion of John and Travis was to create a new-style package with the least effort. I.e., take the current sources contained in the spkg and post them somewhere (aka "upstream"), and install new-stylish from there.
Or perhaps a little better and still feasible: Use the proper existing upstream for MeatAxe (even though the package uses a very old version), and only put the rest of the sources to a new upstream.
In the latter case, the ticket should be set back to needs_work.
OK.
Work Issues: Create a new-style package with least effort
I have moved the documentation of the package. In case you are interested in testing an old-style version 2.1.5, see http://users.minet.uni-jena.de/cohomology/documentation/#installation
Next, I plan to create a least-effort new-style version 2.1.6. It will be basically identical with 2.1.5, except for the package style. And changing everything to the latest MeatAxe upstream will be considerable work taking some time, justifying a new ticket and a major new version 3.0.
Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.
Sigh. Apparently there is also a change in the pickling of Singular objects that I am responsible for and that I have now to cope with...
Replying to @simon-king-jena:
Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.
I guess it's because of the Cython upgrade. Earlier versions of Cython did not give an error for code like
f(foo="hello", foo="world")
but now that's an error just like in Python.
Replying to @jdemeyer:
I guess it's because of the Cython upgrade.
Probably. That error (somewhere doing mutable = True, mutable = True
) is fixed. But the Singular pickling error is likely to be tricky.
Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.
In the past, p_group_cohomology-2.1.4 was an optional spkg. Sage dropped support for old style spkgs, and thus only an experimental spkg remained.
The purpose of this ticket is to provide a new group cohomology package for Sage. I suggest to make it optional (not experimental).
In the experimental version 2.1.5, I have improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.
Package structure
Version 3.0 is refactored as follows:
SAGE_SHARE/
, where GAP and Singular libraries belong.Further changes
sage -t
is enough.Many tests have changed, since the ring presentations of the computed cohomology rings, specifically for non-primepower groups, have changed. I do not fully understand why they changed. but it seems that it is a consequence of changes in Singular. The changes did (of course...) not affect the isomorphism types of the rings.
Upstream tar ball
https://users.fmi.uni-jena.de/cohomology/p_group_cohomology-3.0.tar.xz
Upstream: None of the above - read trac for reasoning.
CC: @jhpalmieri @vbraun @jdemeyer @haraldschilly @frederichan-IMJPRG david.green@uni-jena.de
Component: packages: optional
Keywords: group cohomology
Author: Simon King
Branch:
6576915
Reviewer: Travis Scrimshaw, Jeroen Demeyer
Issue created by migration from https://trac.sagemath.org/ticket/18514