Group cohomology spkg, version 2.1.2

[simon-king-jena]

Version 2.1.2 of the modular group cohomology package is now available. There is extensive documentation. The package needs the Small Groups library (available in the gap-packages spkg) and can be installed with sage -i http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.2.spkg

Most importantly: The version 2.0, that is the current "official" version of the optional package, would not work with the current version of Sage. So, there has to be an upgrade of the spkg.

Moreover, it has new features.

New Features

1. Improved code quality

• IIRC, one of the referee's complaint on earlier versions of the package was that the !init!.py file actually contains code, rather than just importing things. So, I made a new module "factory.py". Now, the !init!.py only contains documentation and import statements.

• Moreover, I split the !call! method of the cohomology ring constructor into smaller units.

• Virtually all computations with this package require creating certain data files on disk. However, the data that are shipped by the package are installed in SAGE_DATA. So, if a user has no write access in that folder, he can read from the database, but can not add further computations to it. In previous versions, it meant that the user had to create his/her own database from scratch. This is solved in version 2.1.1: Still, the user will create his/her own database, but this database can make full use of the stuff that already is in SAGE_DATA.

• Typically one has methods whose result can be cached and needs only be re-computed if the ring approximation changes; this is now being taken care of by decorators.

• I tried to reduce compiler warnings.

• Using the package, several bugs in Singular and GAP have been uncovered. The new Singular version 3-1-1 fixes some bugs, so that in some cases more efficient methods can be used. The package tests the available Singular version; it both supports the new features and is able to work around the old Singular bugs.

• We finally achieved 100% doctest coverage!!

• By oversight, the old test script failed to execute the tests of special methods (such as _add_ or _mul_. The new script now captures all tests, and complains if a class, method or function does not appear to be tested. Also, the test script does parallel tests.

• The parallel testscript uncovered a problem with temporary files associated with Sage's pexpect interfaces -- see #10004. The testscript tests if the problem reported at #10004 is fixed, and will only do parallel testing if it is fixed.

2. Extended computational capability

• The completeness criteria are further improved. We use three criteria, namely the modified Benson criterion (found by David Green and myself), the Symonds criterion and the Hilbert-Poincaré criterion (suggested by Peter Symonds and worked out by myself). Due to new methods of constructing parameters, the Symonds criterion is now often the best choice - but not always, so that it is good to have three methods to choose from.

• New functionality includes the computation of preimages of induced homomorphisms, essential ideals and depth essential ideals. Using the package, a new example of a group was found for which the square of the essential ideal does not vanish (this is only the second known example!). So far, a conjecture of Jon Carlson on depth essential ideals can be confirmed.

3. Portability

The package fully works on t2! This is a very non-trivial achievement. It involved:

• Changing the names of several functions from C-MeatAxe, since the Sun linker seems to confuse them with functions from pari. This is a problem similar to the one found at #1396.
• t2 is a big-endian machine. GAP's random generator depends on the endianness (which is a bug). Certain internal data used by the cohomology computations are obtained by randomised algorithms of GAP. The ring presentation found for the cohomology ring depends on these internal data. So, in order to get machine independent computationally well defined results, special care was needed, as discussed on sage-devel

Testing

As usual, if the environment variable SAGE_CHECK is exported and has the value yes, the test script is automatically executed and the result saved in SAGE_ROOT/install.log.

The script tests parallely. By default, the number of threads is one third of the number of available CPUs. This can be overruled by exporting the environment variable SAGE_NUMBER_THREADS.

However, problems with parallel testing in Version 2.1 revealed a problem with Sage's pexpect interfaces: All interfaces (e.g., GAP and Singular) and all parallel branches use a single temporary file for passing long commands. A solution is at #10004. The test script of the cohomology spkg will only do parallel testing if gap and singular use different temporary files.

I successfully installed and tested the package on the following machines and the following versions of Sage and Singular:

• t2.math, Sage 4.5.1, Singular 3-1-1
• bsd.math, Sage 4.5.2, Singular 3-1-0, both in 64 and 32 bit mode
• sage.math, Sage 4.5.1, Singular 3-1-0
• x86_64 GNU/Linux, Intel Core2 CPU 6700 @ 2.66GHz, Sage 4.5.2, Singular 3-1-1 -- without write permission in SAGE_DATA.

On another GNU Linux machine with AMD processors, it builds and seems to work even with Sage 4.2.1, but apparently the interface to the @parallel decorator has changed, so that I was not able to run the test script.

optional spkg: http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.2.spkg

CC: graham.ellis@nuigalway.ie david.green@uni-jena.de

Component: packages: optional

Keywords: modular group cohomology solaris t2

Author: Simon King

Reviewer: Karl-Dieter Crisman, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/9894

[kcrisman]

comment:1

On a Mac OS X 10.4 G4 PPC running at 1.25 GHz, 1 GB memory, I get successful installation and mostly passed doctests, but some failure. See the relevant bits of the log. I didn't post the whole log because there are some non-failures reported as failures due to another spkg update - I checked these individually, and they are definitely ok. I'm going to try ./sage -ba and reinstall just to make sure, but it's apparent that at least a few of these things are real differences - different list entries, different orderings, etc. Happy hunting!

[simon-king-jena]

comment:2

Hi!

Thank you for testing!

I found this error particularly interesting, at the very end of the file

    sage: H.sylow_cohomology().Dickson
Expected:
    ['-b_2_0^6 - b_2_2^6 - b_2_2^2*b_4_6^2 + b_2_2^3*c_6_11 + b_4_6^3', None]
Got:
    ['b_4_6^3-b_2_2^2*b_4_6^2-b_2_2^6-b_2_0^6+b_2_2^3*c_6_11', None]

What you got is, as much as I see, the result that you would get when computing H.sylow_cohomology() from scratch. But in fact it is expected that H.sylow_cohomology() is automatically downloaded from a web repository, containing cohomology rings that were computed with an old version of the package and thus have equivalent data (note that the polynomials above are equal) presented in a different form.

Could it be that you worked without internet connection? At least, it would explain that particular error.

Often there is a "list index out of range" in MODCOHO.find_relations. This one is a bit obscure to me at the moment. Can you try one of these examples manually and post the full traceback?

There is one error in "BarCode.show"; I have seen this on OS X before, but this seems to be a problem with "show" on OS X in general.

Could you please post the entire log? It seems that you cut off too much. E.g., I can not see how COCH.massey_power fails.

Best regards,

Simon

[simon-king-jena]

Work Issues: tests should work independent of the data the user has on his/her system

[simon-king-jena]

comment:3

Hi!

It seems that there is a problem I hadn't thought of: If one has used the spkg before, then data are accumulated that are later used when doing related computations. Of course, I did use the package before. Now, when testing, it can happen that the old results (potentially obtained with an old program version) are used and spoil the tests.

So, @kcrisman, this might be another explanation for the "particularly interesting" failure: There are old data on my computer, and I accidentally used the old data in the tests.

I have to think how to solve that problem. Certainly the tests should work both with and without having old data in the system. I did try to write the tests so that the computations are either done from scratch or use data that I have full control of, but it seems that I missed some exceptions.

Bets regards, Simon

[kcrisman]

comment:4

I did ./sage -ba to get rid of the erroneous errors (that would include the file you refer to). I'll upload an updated version of that part of the log. Also, there was internet connection (had to have it to download it, and also to get the gap package I didn't realize I needed until it told me so), but it is true I have never computed these before or used the spkg before. As far as I can tell, the second run had the same failures, but I didn't bother to check all of them by hand.

Finally, here is one of the typical failures replicated. I can check some other stuff (possibly not until rather later, though) if you give very explicit instructions as to what commands to run.

Crismans-Computer:~ crisman$ Desktop/sage-4.5.3/sage
----------------------------------------------------------------------
| Sage Version 4.5.3, Release Date: 2010-09-04                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: from pGroupCohomology import CohomologyRing
sage: tmp = tmp_filename()
sage: CohomologyRing.set_user_db(tmp)
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing(G,prime=2,GroupName='A8')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/Users/crisman/<ipython console> in <module>()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/__init__.pyc in __call__(self, *args, **kwds)
   2676             # By now, we have both subgroups and their cohomology rings.
   2677             if not HP.completed:
-> 2678                 HP.make()
   2679             # not needed for HSyl, since it was computed when we did it in the computation of HP
   2680 

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.make (pGroupCohomology/modular_cohomology.c:46208)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.next (pGroupCohomology/modular_cohomology.c:45211)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/cohomology.so in pGroupCohomology.cohomology.COHO.lift_dickson (pGroupCohomology/cohomology.c:63567)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.PrescribedRestrictions (pGroupCohomology/modular_cohomology.c:35570)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.decomposable_classes (pGroupCohomology/modular_cohomology.c:33350)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.find_relations (pGroupCohomology/modular_cohomology.c:32768)()

IndexError: list index out of range

[kcrisman]

comment:5

If there are tests that need internet, by the way, those should be marked # optional - internet.

[simon-king-jena]

comment:6

Replying to @kcrisman:

I did ./sage -ba to get rid of the erroneous errors (that would include the file you refer to). I'll upload an updated version of that part of the log. Also, there was internet connection (had to have it to download it, and also to get the gap package I didn't realize I needed until it told me so), but it is true I have never computed these before or used the spkg before.

Yes, probably that's why. I moved my old data out of the way, and then I got a couple of errors. It turned out that this could be resolved by doing CohomologyRing.user_db(...) when I did CohomologyRing(...) (the latter did access old data).

As far as I can tell, the second run had the same failures, but I didn't bother to check all of them by hand.

Yes, no surprise. My erroneous tests assumed that you had data on your computer obtained by previous versions of the package.

Finally, here is one of the typical failures replicated. I can check some other stuff (possibly not until rather later, though) if you give very explicit instructions as to what commands to run.

Thank you! I'll see if I can somehow replicate the error.

Does the error also occur if you do

sage: from pGroupCohomology import CohomologyRing
sage: tmp = tmp_filename()
sage: CohomologyRing.set_user_db(tmp)
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing.user_db(G,prime=2,GroupName='A8')

?

Best regards, Simon

[simon-king-jena]

comment:7

Replying to @kcrisman:

If there are tests that need internet, by the way, those should be marked # optional - internet.

Thank you! I didn't know that this is possible.

Best regards,

Simon

[simon-king-jena]

comment:8

Replying to @simon-king-jena:

Replying to @kcrisman:

If there are tests that need internet, by the way, those should be marked # optional - internet.

Thank you! I didn't know that this is possible.

However, there are only two tests for which the internet accessibility should really make a difference: There, CohomologyRing.web_db(...) is explicitly called.

When doing CohomologyRing(...) or CohomologyRing.user_db(...), it is tried to download the result from the web repository (unless requested otherwise by the optional argument websource=False), and if this fails, the computation is simply started from scratch, without raising an error.

I am now trying to change the tests so that they will work both with or without old data being present.

[kcrisman]

comment:9

Does the error also occur if you do

sage: from pGroupCohomology import CohomologyRing
sage: tmp = tmp_filename()
sage: CohomologyRing.set_user_db(tmp)
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing.user_db(G,prime=2,GroupName='A8')

It's slightly different:

Crismans-Computer:~ crisman$ Desktop/sage-4.5.3/sage
----------------------------------------------------------------------
| Sage Version 4.5.3, Release Date: 2010-09-04                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: sage: from pGroupCohomology import CohomologyRing
sage:  sage: tmp = tmp_filename()
sage:  sage: CohomologyRing.set_user_db(tmp)
sage:  sage: G = gap('AlternatingGroup(8)')
sage:  sage: H = CohomologyRing.user_db(G,prime=2,GroupName='A8')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/Users/crisman/<ipython console> in <module>()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/__init__.pyc in user_db(self, *args, **kwds)
   2991         """
   2992         kwds['root'] = COHO.user_db
-> 2993         return self(*args, **kwds)
   2994 
   2995     def set_web_db(self, s = None):

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/__init__.pyc in __call__(self, *args, **kwds)
   2676             # By now, we have both subgroups and their cohomology rings.
   2677             if not HP.completed:
-> 2678                 HP.make()
   2679             # not needed for HSyl, since it was computed when we did it in the computation of HP
   2680 

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.make (pGroupCohomology/modular_cohomology.c:46208)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.next (pGroupCohomology/modular_cohomology.c:45211)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/cohomology.so in pGroupCohomology.cohomology.COHO.lift_dickson (pGroupCohomology/cohomology.c:63567)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.PrescribedRestrictions (pGroupCohomology/modular_cohomology.c:35570)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.decomposable_classes (pGroupCohomology/modular_cohomology.c:33350)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.find_relations (pGroupCohomology/modular_cohomology.c:32768)()

IndexError: list index out of range

But essentially the same. Apparently the find_relations method is assuming some list has the wrong length.

[simon-king-jena]

comment:10

Replying to @kcrisman:

Finally, here is one of the typical failures replicated. I can check some other stuff (possibly not until rather later, though) if you give very explicit instructions as to what commands to run.

There are various lists occuring inside find_relations. There is also some Singular commands used to determine coefficients of lists, so, the error could be there as well.

Since I can not replicate it on my computer, can you please do/answer the following?

What version of Sage are you using? What version of Singular is it?

Remove your "private cohomology database" in order to be sure that there is no debris left. You will probably not mind to do it by rm -r ~/.sage/pGroupCohomology/db/* (or you can move the contents of the folder to a backup location). The folder itself must be present, though.

Do the following in Sage:

sage: from pGroupCohomology import CohomologyRing, _cache
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing(G,prime=2,GroupName='A8')
sage: for X in _cache.values():
....:     print X.autosave_name(), X.completed
....:

This should give something like

/home/king/.sage/pGroupCohomology/db/H6gp1mod2.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/8gp3/H8gp3.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/32gp49/H32gp49.sobj True
/home/king/.sage/pGroupCohomology/db/H192gp1493mod2.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/32gp27/H32gp27.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/2gp1/H2gp1.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/64gp138/H64gp138.sobj True
/home/king/.sage/pGroupCohomology/db/HA8mod2.sobj False
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/16gp14/H16gp14.sobj True
/home/king/SAGE/sage-4.4.2/data/pGroupCohomology/8gp5/H8gp5.sobj True

These are all cohomology rings that one needs to know in order to compute H. But since the construction of H fails for you, the list might be different, and at least the .../HA8mod2.sobj should be missing.

You see that the cohomology rings of prime power groups are in the public database SAGE_ROOT/data/..., whereas the non prime power groups are in your private database ~/.sage/....

Do you get a False after a prime power group in the above list?

According to the traceback you got, I expect that you have a False after H192gp1493mod2.sobj. Can you please post (or send me by e-mail) all files from the above list which are followed by False? Then I can do further debugging with them.

Best regards,

Simon

[simon-king-jena]

comment:11

Hi Karl-Dieter,

There is also another way to potentially get more insight: Could you try the computation in protocol mode, i.e., H = CohomologyRing(G,prime=2,GroupName='A8',options='prot')? The last few protocol lines before the error will probably help to locate the problem.

Best regards, Simon

[simon-king-jena]

Changed work issues from tests should work independent of the data the user has on his/her system to bug in find_relations on some systems; tests should work independent of the data the user has on his/her system

[simon-king-jena]

comment:12

Perhaps I found the solution: In find_relations (and various other places), I parsed an ideal M in Singular into a list of strings as follows:

singular.eval('print(%s)'%M.name()).split(',\n')

I guess that split(',\n') is a mistake: There are systems (including OS X?) on which '\n' is not the line break character. So, I should better do

[t.strip() for t in singular.eval('print(%s)'%M.name()).split(',')]

I am now preparing and testing an updated version. So, until later...

Best regards,

Simon

[simon-king-jena]

Changed work issues from bug in find_relations on some systems; tests should work independent of the data the user has on his/her system to Is the code independent of the computer's newline character?

[simon-king-jena]

comment:13

Hi!

I updated the package, changing the tests so that they should work with or without old data being present, and changing the code so that it is not expected that '\n' is the newline character.

I tested it on the desktop computer in my office and am currently running tests on bsd.math, t2.math and sage.math.

So, it is ready for review again.

If you want to re-install and test the package, please don't forget to delete the copy of the old package version, for otherwise the new version would not be downloaded. Hence,

rm $SAGE_ROOT/spkg/optional/p_group_cohomology-2.1.spkg
sage -f http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.spkg

Testing with

export SAGE_CHECK=yes

Bounding the number of parallel tests with, e.g.

export SAGE_NUMBER_THREADS=2

Cheers,

Simon

[simon-king-jena]

comment:14

There are doctest failures on t2. So, back at work...

[simon-king-jena]

Changed work issues from Is the code independent of the computer's newline character? to Fix problems on t2. Is the code independent of the computer's newline character?

[simon-king-jena]

comment:15

Replying to @simon-king-jena:

There are doctest failures on t2. So, back at work...

But it's strange: All errors come from GAP, and they are like

TypeError: Error evaluating $sage81:=IsBoundGlobal("exportMTXLIB");; in Gap

or

 TypeError: Error evaluating $sage24:="DihedralB";; in Gap

Of course, both commands are supposed to work! I could imagine that it is a parallelisation problem. The errors occurred with SAGE_NUMBER_THREADS=8. I am now seeing if it works with SAGE_NUMBER_THREADS=4.

[simon-king-jena]

Changed work issues from Fix problems on t2. Is the code independent of the computer's newline character? to Is the code independent of the computer's newline character?

[simon-king-jena]

comment:16

Replying to @simon-king-jena:

Replying to @simon-king-jena:

There are doctest failures on t2. So, back at work...

But it's strange: All errors come from GAP...

And when I did it again, they vanished. Strange enough. But I think testing on the machine of kcrisman is more important. Can you please try?

Best regards,

Simon

[kcrisman]

comment:17

And when I did it again, they vanished. Strange enough. But I think testing on the machine of kcrisman is more important. Can you please try?

Eventually, but at least not until tonight, possibly not until several evenings from now. I will try hard to rm all relevant stuff, though, before trying all this out :)

[kcrisman]

comment:18

Since I can not replicate it on my computer, can you please do/answer the following?

What version of Sage are you using? What version of Singular is it?

I have no explanation for the following.

----------------------------------------------------------------------
| Sage Version 4.5.3, Release Date: 2010-09-04                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: singular_version()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/Users/crisman/<ipython console> in <module>()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/sage/interfaces/singular.pyc in singular_version()
   1916     EXAMPLES:
   1917     """
-> 1918     return singular.eval('system("--version");')
   1919 
   1920 

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/sage/interfaces/singular.pyc in eval(self, x, allow_semicolon, strip, **kwds)
    547 
    548         if s.find("error") != -1 or s.find("Segment fault") != -1:
--> 549             raise RuntimeError, 'Singular error:\n%s'%s
    550 
    551         if get_verbose() > 0:

RuntimeError: Singular error:
   ? cannot open `help.cnf`
Singular for ppcMac-darwin version 3-1-1 (3114-2010090802)  Sep  8 2010 02:24:06
with
        factory(@(#) factoryVersion = 3.1.1),libfac(3.1.1,Feb 2010),
        GMP(4.2),NTL(5.4.2),32bit,static readline,Plural,DBM,
        dynamic modules,OM_NDEBUG,random=1284424481
        CC= gcc -O3 -g -fPIC -pipe -DNDEBUG -DOM_NDEBUG -DppcMac_darwin -DHAVE_CONFIG_H,
        CXX= g++ -O3 -g -fPIC -pipe -DNDEBUG -DOM_NDEBUG -DppcMac_darwin -DHAVE_CONFIG_H (4.0.1 (Apple Computer, Inc. build 5370))
argv[0]   :     Singular-3-1-1
SearchPath:     /Users/crisman/Desktop/sage-4.5.3/local/share/singular:/Users/crisman/Desktop/sage-4.5.3/local/LIB
Singular  :     /Users/crisman/Desktop/sage-4.5.3/local/bin/Singular
BinDir    :     /Users/crisman/Desktop/sage-4.5.3/local/bin
RootDir   :     /Users/crisman/Desktop/sage-4.5.3/local
DefaultDir:     /Users/crisman/Desktop/sage-4.5.3/local
InfoFile  :
IdxFile   :
HtmlDir   :
ManualUrl :     http://www.singular.uni-kl.de/Manual/3-1-1
ExDir     :
Path      :     /Users/crisman/Desktop/sage-4.5.3/local/bin:/Users/crisman/Desktop/sage-4.5.3:/bin:/sbin:/usr/bin:/usr/sbin
EmacsDir  :
Available HelpBrowsers: dummy, emacs, 
Current HelpBrowser: dummy 
   ? error occurred in or before STDIN line 49965: `system("--version");`

But as you can see in the error messages, it's the 3.1.1 devel version, and this works:

Crismans-Computer:~ crisman$ Desktop/sage-4.5.3/sage -singular
                     SINGULAR                             /  Development
 A Computer Algebra System for Polynomial Computations   /   version 3-1-1
                                                       0<
     by: G.-M. Greuel, G. Pfister, H. Schoenemann        \   Feb 2010
FB Mathematik der Universitaet, D-67653 Kaiserslautern    \

Remove your "private cohomology database" in order to be sure that there is no debris left. You will probably not mind to do it by rm -r ~/.sage/pGroupCohomology/db/* (or you can move the contents of the folder to a backup location). The folder itself must be present, though.

Do the following in Sage:

sage: from pGroupCohomology import CohomologyRing, _cache
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing(G,prime=2,GroupName='A8')

This fails for me earlier:

sage: from pGroupCohomology import CohomologyRing, _cache
sage: G = gap('AlternatingGroup(8)')
sage: H = CohomologyRing(G,prime=2,GroupName='A8')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/Users/crisman/<ipython console> in <module>()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/__init__.pyc in __call__(self, *args, **kwds)
   2676             # By now, we have both subgroups and their cohomology rings.
   2677             if not HP.completed:
-> 2678                 HP.make()
   2679             # not needed for HSyl, since it was computed when we did it in the computation of HP
   2680 

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.make (pGroupCohomology/modular_cohomology.c:46208)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.next (pGroupCohomology/modular_cohomology.c:45211)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/cohomology.so in pGroupCohomology.cohomology.COHO.lift_dickson (pGroupCohomology/cohomology.c:63567)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.PrescribedRestrictions (pGroupCohomology/modular_cohomology.c:35570)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.decomposable_classes (pGroupCohomology/modular_cohomology.c:33350)()

/Users/crisman/Desktop/sage-4.5.3/local/lib/python2.6/site-packages/pGroupCohomology/modular_cohomology.so in pGroupCohomology.modular_cohomology.MODCOHO.find_relations (pGroupCohomology/modular_cohomology.c:32768)()

IndexError: list index out of range

I'll try the reinstall next.

[kcrisman]

comment:19

Replying to @simon-king-jena:

Hi Karl-Dieter,

There is also another way to potentially get more insight: Could you try the computation in protocol mode, i.e., H = CohomologyRing(G,prime=2,GroupName='A8',options='prot')? The last few protocol lines before the error will probably help to locate the problem.

Cool documentation of what is going on! I removed the db stuff (but not the folder) again, and got this before the same error (I assume that up to here it is the same for you):

There is no new generator of H^*(SmallGroup(192,1493); GF(2)) in degree 7
  Degree 7 of the visible ring structure of H^*(SmallGroup(192,1493); GF(2)) is computed!
  Saving H^*(SmallGroup(192,1493); GF(2))
  We expect that the Hilbert-Poincare criterion will not apply before degree 11
  Start computation in Degree 8
  All stable cocycles are decomposable
  Determine degree 8 standard monomials of H^*(SmallGroup(192,1493); GF(2))
  Express 48 standard monomials as cocycles
  > Interreduction
  > Determining decomposable subspace
  > Extracting relations
  There is no new relation in degree 8
There is no new generator of H^*(SmallGroup(192,1493); GF(2)) in degree 8
  Try to lift 1st power of 0th Dickson invariant
  Determine degree 8 standard monomials of H^*(SmallGroup(192,1493); GF(2))
  Express 48 standard monomials as cocycles
  > Interreduction
  > Determining decomposable subspace
  > Extracting decomposable cocycles and relations
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

[kcrisman]

comment:20

If you want to re-install and test the package, please don't forget to delete the copy of the old package version, for otherwise the new version would not be downloaded. Hence,

rm $SAGE_ROOT/spkg/optional/p_group_cohomology-2.1.spkg
sage -f http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.spkg

Testing with

export SAGE_CHECK=yes

Currently doing this. It will take some time.

export SAGE_NUMBER_THREADS=2

lol - this processor has one core, and you can't do multiple thread with Sage on it (I've tried).

[simon-king-jena]

comment:21

Hi!

Replying to @kcrisman:

... Determining decomposable subspace Extracting decomposable cocycles and relations

Yep. That confirms my conjecture. Right after that line of protocol, there is a line containing a split statement, and IIRC it originally was

SelfValues = singular.eval('print(%s)'%DG.name()).split(',\n')

In the current version, it is

SelfValues = [t.strip() for t in singular.eval('print(%s)'%DG.name()).split(',')]

instead.

Could you test the following, please:

sage: R = singular.ring(0,'(x,y)','dp')
sage: I = singular.maxideal(3)
sage: singular.eval('print(%s)'%I.name()).split(',\n')
['y^3', 'x*y^2', 'x^2*y', 'x^3']
sage: [t.strip() for t in singular.eval('print(%s)'%I.name()).split(',')]
['y^3', 'x*y^2', 'x^2*y', 'x^3']

I reckon that on your computer the two lists are different.

Cheers,

Simon

[kcrisman]

comment:22

Could you test the following, please:

sage: R = singular.ring(0,'(x,y)','dp')
sage: I = singular.maxideal(3)
sage: singular.eval('print(%s)'%I.name()).split(',\n')
['y^3', 'x*y^2', 'x^2*y', 'x^3']
sage: [t.strip() for t in singular.eval('print(%s)'%I.name()).split(',')]
['y^3', 'x*y^2', 'x^2*y', 'x^3']

Well: {{{---------------------------------------------------------------------- | Sage Version 4.5.3, Release Date: 2010-09-04 | | Type notebook() for the GUI, and license() for information. |

---

sage: R = singular.ring(0,'(x,y)','dp') sage: I = singular.maxideal(3) sage: singular.eval('print(%s)'%I.name()).split(',\n') ['y^3', 'xy^2', 'x^2y', 'x^3'] sage: [t.strip() for t in singular.eval('print(%s)'%I.name()).split(',')] ['y^3', 'xy^2', 'x^2y', 'x^3'] }}} But:

Successfully installed p_group_cohomology-2.1
Running the test suite.
Testing the package pGroupCohomology...
Will use 1 parallel processes

pGroupCohomology fails
pGroupCohomology.unit_test_64 passes in 585.23 s
pGroupCohomology._IsKeyEquivalent passes in 52.95 s
<snip>
pGroupCohomology.resolution.G_ALG.kG_map passes in 16.91 s
pGroupCohomology.resolution.G_ALG.isAbelian passes in 17.09 s
pGroupCohomology.resolution.MasseyDefiningSystems passes in 15.39 s
pGroupCohomology.resolution.MasseyDefiningSystems.__init__ passes in 15.38 s
pGroupCohomology.resolution.MasseyDefiningSystems._make passes in 15.43 s
pGroupCohomology.resolution.MasseyDefiningSystems.values passes in 17.53 s
There were doctest failures:
pGroupCohomology:
sage -t -optional -long "/Users/crisman/.sage/temp/Crismans_Computer.local/3200/dir_0/file_0.py"
*** *** Error: TIMED OUT! PROCESS KILLED! *** ***

         [1803.1 s]

----------------------------------------------------------------------
The following tests failed:

        sage -t -optional -long "/Users/crisman/.sage/temp/Crismans_Computer.local/3200/dir_0/file_0.py" # Time out
Total time for all tests: 1803.4 seconds

Summary
-------

The following tests fail:
  pGroupCohomology
Total time: 15702.23 sec

So now I'll just

export SAGE_TIMEOUT_LONG=3600

and ./sage -f it again, and hopefully all will be well in the morning. I have to say, five hours to install and test - that's quite an spkg, quite the computation!

[kcrisman]

comment:23

Not sure what happened there - must have formatted wrong.

----------------------------------------------------------------------
| Sage Version 4.5.3, Release Date: 2010-09-04                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: R = singular.ring(0,'(x,y)','dp')
sage: I = singular.maxideal(3)
sage: singular.eval('print(%s)'%I.name()).split(',\n')
['y^3', 'x*y^2', 'x^2*y', 'x^3']
sage: [t.strip() for t in singular.eval('print(%s)'%I.name()).split(',')]
['y^3', 'x*y^2', 'x^2*y', 'x^3']

[kcrisman]

comment:24

If that fails again, I would like to know where these doctests live in the spkg so I can figure out how to run them individually.

[simon-king-jena]

comment:25

Replying to @kcrisman:

and ./sage -f it again, and hopefully all will be well in the morning. I have to say, five hours to install and test - that's quite an spkg, quite the computation!

Ouch! Five hours is really much. On my desktop computer, the total time for the tests is 2812.09 sec, and installation takes maybe 10 minutes.

By the way, the test skript runs all doctests. So, it always includes the ones marked long. Perhaps in the next version I should provide the possibility to only run the short tests.

But what is the point of having tests that take so long? This is due to the fact that often I was not able to find smaller examples that show the benefit of a particular method.

How to find the doctests? Well, you had a timeout in pGroupCohomology. So, you are in the lucky situation that the test is in a Python file, namely (if you untar'd the package) in p_group_cohomology-2.1/src/pGroupCohomology/__init__.py. You could run sage -t -verbose -long on that file (that wouldn't work for the Cython files). Using the "verbose" option would show exactly what part of the test takes so long.

It is surprising to me that the two lists in the little split test are equal. But anyway, I think [t.strip() for t in singular.eval('print(%s)'%I.name()).split(',')] is cleaner than singular.eval('print(%s)'%I.name()).split(',\n')

Cheers,

Simon

[simon-king-jena]

comment:26

Replying to @kcrisman:

I have to say, five hours to install and test - that's quite an spkg, quite the computation!

PS:

On my machine, the longest single test takes about 90 seconds. Since the documentation of pGroupCohomology.__init__ contains many examples (it is an introduction how to use the package), it is no surprise that it takes longer, but it is still pGroupCohomology passes in 242.05 s.

[kcrisman]

Changed work issues from Is the code independent of the computer's newline character? to none

[kcrisman]

Reviewer: Karl-Dieter Crisman

[kcrisman]

comment:27

On my machine, the longest single test takes about 90 seconds. Since the documentation of pGroupCohomology.__init__ contains many examples (it is an introduction how to use the package), it is no surprise that it takes longer, but it is still pGroupCohomology passes in 242.05 s.

pGroupCohomology passes in 1205.79 s

So less than six times slower - not bad ;)

I think the timeout was because I had a heavy other load (watching Carl Witty's video, downloading things) when I ran it the first time. But remember, this is a 1.25 GHz machine with only 1 GB memory!

Anyway, I can't review this any more because I don't use Singular or GAP much at all (though I still think it's cool that you are calculating the cohomology rings as rings, there has to be a way to apply this nicely to the topology stuff in Sage), but I will at least add myself as a first reviewer.

[simon-king-jena]

comment:28

Replying to @kcrisman:

I think the timeout was because I had a heavy other load (watching Carl Witty's video, downloading things) when I ran it the first time. But remember, this is a 1.25 GHz machine with only 1 GB memory!

OK, my machine has 2 GB. But actually keeping the memory consumption low was a driving force behind many things that I did for the first version of the package. As a result, the cohomology for most (but not all) groups of order 128 can be computed with 2 GB.

But I think I should put "find smaller test cases" on my to-do list for version 2.2.

Anyway, I can't review this any more because I don't use Singular or GAP much at all

What a pity! But thanks for your effort!

(though I still think it's cool that you are calculating the cohomology rings as rings, there has to be a way to apply this nicely to the topology stuff in Sage),

I wonder if this will be possible. As it is, the package is very much addressed to algebra, and I don't see how this could be changed.

but I will at least add myself as a first reviewer.

Sure!

Best regards,

Simon

[simon-king-jena]

comment:29

In the meantime, I created a new version of the package (and changed the description accordingly). It was inspired by David Green's complaint that the package would try to load data from SAGE_DATA but would then be unable to do further computations without write permission.

[simon-king-jena]

Description changed:

--- 
+++ 
@@ -1,25 +1,39 @@
-Version 2.1 of the modular group cohomology package is now available. There is extensive [documentation](http://sage.math.washington.edu/home/SimonKing/Cohomology/index.html). The package can be installed with `sage -i http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.spkg`
+Version 2.1.1 of the modular group cohomology package is now available. There is extensive [documentation](http://sage.math.washington.edu/home/SimonKing/Cohomology/index.html). The package needs the Small Groups library (available in the gap-packages spkg) and can be installed with `sage -i http://sage.math.washington.edu/home/SimonKing/Cohomology/p_group_cohomology-2.1.1.spkg`

 **__New Features__**

-1. I tried to improve the code quality. For example:
+**1. Improved code quality**

-   - Typically one has methods whose result can be cached and needs only be re-computed if the ring approximation changes; this is now being taken care of by decorators.
-   - I tried to reduce compiler warnings.
-   - Using the package, several bugs in Singular and GAP have been uncovered. The new Singular version 3-1-1 fixes some bugs, so that in some cases more efficient methods can be used. The package tests the available Singular version; it both supports the new features and is able to work around the old Singular bugs.
+  - IIRC, one of the referee's complaint on earlier versions of the package was that the !__init!__.py file actually contains code, rather than just importing things. So, I made a new module "factory.py". Now, the !__init!__.py only contains documentation and import statements.

-2. The completeness criteria are further improved. We use three criteria, namely the modified Benson criterion (found by David Green and myself), the Symonds criterion and the Hilbert-Poincaré criterion (suggested by Peter Symonds and worked out by myself). Due to new methods of constructing parameters, the Symonds criterion is now often the best choice - but not always, so that it is good to have three methods to choose from.
+  - Moreover, I split the !__call!__ method of the cohomology ring constructor into smaller units.

-3. New functionality includes the computation of preimages of induced homomorphisms, essential ideals and depth essential ideals. Using the package, a new example of a group was found for which the square of the essential ideal does not vanish (this is only the second known example!). So far, a conjecture of Jon Carlson on depth essential ideals can be confirmed.
+  - Virtually all computations with this package require creating certain data files on disk. However, the data that are shipped by the package are installed in SAGE_DATA. So, if a user has no write access in that folder, he can read from the database, but can not add further computations to it. In previous versions, it meant that the user had to create his/her own database *from scratch*. This is solved in version 2.1.1: Still, the user will create his/her own database, but this database can make full use of the stuff that already is in SAGE_DATA.

-4. **It works on t2!! ** This is a very non-trivial achievement. It involved:
-   * Changing the names of several functions from `C-MeatAxe`, since the Sun linker seems to confuse them with functions from pari. This is a problem similar to the one found at #1396.
-   * t2 is a big-endian machine. GAP's random generator depends on the endianness (which is a bug). Certain internal data used by the cohomology computations are obtained by randomised algorithms of GAP. The ring presentation found for the cohomology ring depends on these internal data. So, in order to get machine independent computationally well defined results, special care was needed, as discussed on [sage-devel](http://groups.google.com/group/sage-devel/browse_thread/thread/623f7291ab7e782e)
+  - Typically one has methods whose result can be cached and needs only be re-computed if the ring approximation changes; this is now being taken care of by decorators.

-5. By oversight, the old test script failed to execute the tests of special methods (such as `__add__` or `__mul__`. The new script now captures all tests, and complains if a class, method or function does not appear to be tested. Also, the test script does parallel tests.
+  - I tried to reduce compiler warnings.

-6. We finally achieved **100% doctest coverage!! **
+  - Using the package, several bugs in Singular and GAP have been uncovered. The new Singular version 3-1-1 fixes some bugs, so that in some cases more efficient methods can be used. The package tests the available Singular version; it both supports the new features and is able to work around the old Singular bugs.
+
+  - We finally achieved **100% doctest coverage!! **
+
+  - By oversight, the old test script failed to execute the tests of special methods (such as `__add__` or `__mul__`. The new script now captures all tests, and complains if a class, method or function does not appear to be tested. Also, the test script does parallel tests.
+
+  - The parallel testscript uncovered a problem with temporary files associated with Sage's pexpect interfaces -- see #10004. The testscript tests if the problem reported at #10004 is fixed, and will only do parallel testing if it is fixed.
+
+**2. Extended computational capability**
+
+  - The completeness criteria are further improved. We use three criteria, namely the modified Benson criterion (found by David Green and myself), the Symonds criterion and the Hilbert-Poincaré criterion (suggested by Peter Symonds and worked out by myself). Due to new methods of constructing parameters, the Symonds criterion is now often the best choice - but not always, so that it is good to have three methods to choose from.
+
+  - New functionality includes the computation of preimages of induced homomorphisms, essential ideals and depth essential ideals. Using the package, a new example of a group was found for which the square of the essential ideal does not vanish (this is only the second known example!). So far, a conjecture of Jon Carlson on depth essential ideals can be confirmed.
+
+**3. Portability **
+
+The package fully works on t2! This is a very non-trivial achievement. It involved:
+* Changing the names of several functions from `C-MeatAxe`, since the Sun linker seems to confuse them with functions from pari. This is a problem similar to the one found at #1396.
+* t2 is a big-endian machine. GAP's random generator depends on the endianness (which is a bug). Certain internal data used by the cohomology computations are obtained by randomised algorithms of GAP. The ring presentation found for the cohomology ring depends on these internal data. So, in order to get machine independent computationally well defined results, special care was needed, as discussed on [sage-devel](http://groups.google.com/group/sage-devel/browse_thread/thread/623f7291ab7e782e)

 **__Testing__**

@@ -27,10 +41,14 @@

 The script tests parallely. By default, the number of threads is one third of the number of available CPUs. This can be overruled by exporting the environment variable `SAGE_NUMBER_THREADS`.

+However, problems with parallel testing in Version 2.1 revealed a problem with Sage's `pexpect` interfaces: *All* interfaces (e.g., GAP and Singular) and all parallel branches use a single temporary file for passing long commands. A solution is at #10004. The test script of the cohomology spkg will only do parallel testing if gap and singular use different temporary files.
+
+
 I successfully installed and tested the package on the following machines and the following versions of Sage and Singular:
 * t2.math, Sage 4.5.1, Singular 3-1-1
 * bsd.math, Sage 4.5.2, Singular 3-1-0, both in 64 and 32 bit mode
 * sage.math, Sage 4.5.1, Singular 3-1-0
-* x86_64 GNU/Linux, Intel Core2 CPU 6700 @ 2.66GHz, Sage 4.5.2, Singular 3-1-1
+* x86_64 GNU/Linux, Intel Core2 CPU 6700 @ 2.66GHz, Sage 4.5.2, Singular 3-1-1 -- *without* write permission in SAGE_DATA.

 On another GNU Linux machine with AMD processors, it builds and seems to work even with Sage 4.2.1, but apparently the interface to the `@`parallel decorator has changed, so that I was not able to run the test script.
+

[kcrisman]

comment:30

What do you think needs still to be done/checked for positive review?

[simon-king-jena]

comment:31

Replying to @kcrisman:

What do you think needs still to be done/checked for positive review?

I think the guide line are the new features, which are (according to the project home page):

• v2.1.1: Usage of symbolic links to the public database, so that one can use (but of course not install) this package even without write permission in SAGE_DATA. Restructuring the code. Parallel testing is now only permitted if ticket #10004 is applied (September 2010).
• v2.1: Support for big endian machines. 100% doctest coverage, parallel testing. New: Essential and depth essential ideals. Improved completion tests. (September 2010).

In other words, it would be good if someone who is not author (i.e., not I) does parallel tests on a machine like T2 (which is big endian), in the following setting: The package is installed by someone with write permission in SAGE_DATA, but tested by someone without write permission in SAGE_DATA.

And from the mathematical point of view, one might look at essential and depth essential ideals.

[kcrisman]

comment:32

Hmm, I don't know how to do # 1 (though I have access to a big endian machine), since I am the only user on the box; I definitely am not enough of a group cohomology expert to do # 2!

[simon-king-jena]

comment:33

Replying to @kcrisman:

Hmm, I don't know how to do # 1 (though I have access to a big endian machine), since I am the only user on the box; I definitely am not enough of a group cohomology expert to do # 2!

I did it like that: I installed Sage plus the database_gap package plus the cohomology package as root, and then I did computations in my usual account.

Another possibility: Install the package in the usual way, but then change the permissions (recursively, of course) for SAGE_DATA, so that you have read but not write permission - then you should still be able to use the package.

[simon-king-jena]

comment:34

-- bump --

Any reviewer for version 2.1.1 of the optional group cohomology package, after 6 months?

[kcrisman]

comment:35

I wish I could. It seemed to work great at the time, but I just don't have the time to do the things needed to test some of the things you mention above. It does seem a big shame it's not positively reviewed.

[jhpalmieri]

comment:36

I tried this out with SAGE_CHECK=yes, and I'm getting doctest failures on three different platforms: sage.math, OS X, and fulvia (a skynet machine which is OpenSolaris on x86). I'm attaching logs for them. On sage.math, this was based on a vanilla copy of 4.7.1.alpha4, and on the other machines, it was on top of 4.7.1.rc0. I'm also building on the skynet machine mark, but it's abominably slow, so I don't know when it will finish.

Do I need to be concerned about these?

[jhpalmieri]

Attachment: sage-math.log

doctest failures, sage.math.washington.edu

[jhpalmieri]

doctest failures, OS X box

[jhpalmieri]

Attachment: osx.log

Attachment: fulvia-opensolaris.log

doctest failures, OpenSolaris

[simon-king-jena]

comment:37

Hi John!

Replying to @jhpalmieri:

I tried this out with SAGE_CHECK=yes, and I'm getting doctest failures on three different platforms: sage.math, OS X, and fulvia

Thank you for returning to this!

It is rather unfortunate to have failures. I hope that I will have time to fix them -- I am already in the middle of the next project.

I'm also building on the skynet machine mark, but it's abominably slow, so I don't know when it will finish.

Do I need to be concerned about these?

I guess if there are failures on three different machines then you don't need to wait for mark.

One question: Most of the errors look like

    sig_on_count()
Expected:
    0
Got:
    15

What does that sig_on_count() mean? Does that mean that I forgot _sig_off after _sig_on?

[jhpalmieri]

comment:38

What does that sig_on_count() mean? Does that mean that I forgot _sig_off after _sig_on?

I have no idea, but it sounds like a good guess to me.

[simon-king-jena]

Work Issues: fix doc tests