sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.47k stars 487 forks source link

Upgrade to GAP 4.10 #22626

Closed nthiery closed 5 years ago

nthiery commented 7 years ago

GAP 4.10 comes with a completely rewritten build system that will simplify our packaging. In particular, libGAP no longer needs to be a separate package.

What the branch does:

Status

Currently broken - crashes deep inside GAP error handling system after few simple commands.

Basic tests on libgap:

sage: libgap.eval("GAPInfo.Version")
sage: libgap.DihedralGroup(10).CharacterTable()
CharacterTable( <pc group of size 10 with 2 generators> )
sage: libgap.Group(libgap.eval("[(1,2,3),(1,2)]")).Size()
6

Running most relevant tests:

sage -tp 8 sage/groups sage/libs/gap

Current status: lots of errors

Testing packages with dynamic loading (e.g. IO):

Install IO:

cd $SAGE_LOCAL/share/gap/pkg
wget https://www.gap-system.org/pub/gap/gap4/tar.gz/packages/io-4.5.4.tar.gz
tar xvf io-4.5.4.tar.gz
cd io-4.5.4
./configure --with-gaproot=../..
make

Test it locally:

cd $SAGE_ROOT
gap
gap> LoadPackage("IO");
true

This does not yet work:

#W dlopen() error: /home/embray/src/sagemath/sage/local/share/gap/pkg/io-4.5.4/bin/x86_64-pc-linux-gnu-default64/io.so: undefine\
d symbol: ChangedBags
Error, module '/home/embray/src/sagemath/sage/local/share/gap/pkg/io-4.5.4/bin/x86_64-pc-linux-gnu-default64/io.so' not found
Error, was not in any namespace
Syntax warning: Unbound global variable in /home/embray/src/sagemath/sage/local/share/gap/lib/init.g:803
    ColorPrompt( UserPreference( "UseColorPrompt" ) );
               ^
Error, SetGasmanMessageStatus: function is not yet defined
true

This should be fixed once GAP's gap binary is built on top of libgap. See: https://github.com/markuspf/gap/issues/1 I believe this is fixed, but there are still some problems with the way this ticket is "installing" GAP for $SAGE_LOCAL.

Note:

Upstream PRs

A few pull requests we made to improve use of GAP as a library:

Other open issues that don't have PRs yet:

Some other PRs useful to this effort:

The update of gap_packages and database_gap is on #26856

Tarball: https://www.gap-system.org/pub/gap/gap-4.10/tar.gz/gap-4.10.0.tar.gz

Depends on #26874

CC: @alex-konovalov @dimpase @embray @kiwifb @antonio-rojas @sebasguts @jpflori @sagetrac-markuspf @nthiery @slel @vbraun @williamstein @timokau

Component: interfaces

Keywords: days85, libgap

Stopgaps: error handling in libgap, documentation display

Author: Nicolas M. Thiéry, Dima Pasechnik, Erik Bray, Jeroen Demeyer

Branch: b446ebb

Reviewer: Erik Bray, Dima Pasechnik, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/22626

jdemeyer commented 6 years ago
comment:45

Replying to @dimpase:

there is also libgap unprefixing in Python/Cython that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.

Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.

dimpase commented 6 years ago
comment:46

Replying to @jdemeyer:

Replying to @dimpase:

there is also libgap unprefixing in Python/Cython that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.

Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.

I am not sure I understand. Currently in Cython on can do something like

    from sage.libs.gap.libgap import libgap
    g=libgap.ProjectiveGeneralLinearGroup(3,3)

How will this change under what you propose?

jdemeyer commented 6 years ago
comment:47

Replying to @dimpase:

    from sage.libs.gap.libgap import libgap
    g=libgap.ProjectiveGeneralLinearGroup(3,3)

How will this change under what you propose?

The Python interface won't change at all. I am talking about how sage.libs.gap calls libGAP.

jdemeyer commented 6 years ago
comment:48

As far as I can tell, GAP 4.9 has not been released. Does anybody know how close we are to an actual release?

slel commented 6 years ago
comment:49

I think GAP 4.9.0 is considered a public beta for GAP 4.9.

The list of releases at

still displays GAP 4.8.10 as the latest release.

In the list of past releases there, the GAP 4.8, 4.7, 4.6, 4.5 series start at GAP 4.8.2, 4.7.2, 4.6.2, 4.5.4, so probably GAP 4.x.y with those x and lower y were considered beta.

Still, seeing how 4.9.0 works with Sage would be nice.

jdemeyer commented 6 years ago

Dependencies: #25273

jdemeyer commented 6 years ago
comment:51

Where is the GAP source tarball which is supposed to be used here?

jdemeyer commented 6 years ago

Changed branch from u/nthiery/upgrade_to_gap_4_9 to u/jdemeyer/upgrade_to_gap_4_9

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from 7c04025 to b7278a1

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

72b5066Unprefixed libGAP interface
b7278a1Upgrade to GAP 4.9
dimpase commented 6 years ago
comment:54

Replying to @jdemeyer:

Where is the GAP source tarball which is supposed to be used here?

here: https://www.gap-system.org/pub/gap/gap-4.9/beta/

jdemeyer commented 6 years ago

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@

 The branch attached to this ticket updates Sage to run on top of
 [a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
-by Markus Pfeiffer that adds libgap compilation and [will be merged](https://github.com/fingolfin/gap/pull/64) soon in the devel version of GAP.
+by Markus Pfeiffer that adds libgap compilation and [will be merged](https://github.com/gap-system/gap/pull/1205) soon in the devel version of GAP.

 See https://github.com/markuspf/gap/issues/2 for the few sticking points that could prevent using a vanilla GAP from the distribution (please edit further if you think about more of them).

@@ -13,7 +13,7 @@

 - Update the gap spkg to the new build system and build and install libgap

-- Replace gap.shi.patch by a plain gap startup script for Sage
+- Replace `gap.shi.patch` by a plain gap startup script for Sage

   Rationale: GAP used to provide a startup shell script. The GAP devs
   are in the process of getting rid of it and provide a very minimal
@@ -22,7 +22,7 @@
 - Update a few doctests w.r.t. changes of output of some GAP functions

 - **Possibly controversial:** The new libgap currently *does not come*
-  with symbol rewriting (Foo -> libGAP_Foo). This avoids messing
+  with symbol rewriting (`foobar` -> `libGAP_foobar`). This avoids messing
   around with GAP's sources; in particular opening the door for using
   a stock GAP from the OS distribution. However there always is a risk
   of name conflict. And indeed, GAP's constants (actually cpp macros)
@@ -51,13 +51,10 @@
 TODO:

 - Automatic handling of headers (see below for how to do it by hand).
-  GAP's build system will eventuall provide a rule to install headers
+  GAP's build system will eventually provide a rule to install headers
   which will make this trivial.

 - Use GAP's own `make install` [when it will be implemented](https://github.com/fingolfin/gap/issues/44).
-
-- Update the documentation in sage.libs.gap.libgap.pyx to not mention
-  the `libgap_` prefix

 - Check against #19915 to see if any of the changes there should be
   ported here. Then close as won't fix.
@@ -162,4 +159,3 @@

This should be fixed once GAP's gap binary is built on top of libgap. See: https://github.com/markuspf/gap/issues/1.

jdemeyer commented 6 years ago

Replying to @nthiery:

The branch attached to this ticket updates Sage to run on top of a branch of GAP by Markus Pfeiffer that adds libgap compilation and will be merged soon in the devel version of GAP.

It seems that this branch which "will be merged soon" is still not merged. It would be good to have an idea of whether or not that will happen.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from b7278a1 to 40e644a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d28b575Unprefixed libGAP interface
daf8754Unprefixed libGAP interface
40e644aUpgrade to GAP 4.9
nthiery commented 6 years ago
comment:58

According to https://github.com/gap-system/gap/pull/1205#issuecomment-385670578, Markus's libgap branch won't be merged in GAP 4.9: it's not yet ready enough. Bummer.

We are discussing with Sebastian and Thomas what we can do to move forward here. They are very interested as well by libgap for the integration between GAP and Julia.

Tentative steps:

slel commented 6 years ago
comment:61

GAP 4.9.1 was released in May 2018.

This is the first stable release of GAP 4.9 (GAP 4.9.0 was a beta for GAP 4.9).

slel commented 6 years ago

Description changed:

--- 
+++ 
@@ -1,11 +1,18 @@
-GAP 4.9 will come with a completely rewritten build system that will
-simplify our packaging. In fact, it may well enable Sage to use a vanilla GAP installation as provided by the distribution. It will also enable building GAP as a library, so that we can get rid of our separate libgap package.
+GAP 4.9 comes with a completely rewritten build system that will simplify
+our packaging. In fact, it may well enable Sage to use a vanilla GAP
+installation as provided by the distribution.
+
+The GAP team will not provide libGAP with GAP 4.9, but only with GAP 4.10.

 The branch attached to this ticket updates Sage to run on top of
 [a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
-by Markus Pfeiffer that adds libgap compilation and [will be merged](https://github.com/gap-system/gap/pull/1205) soon in the devel version of GAP.
+by Markus Pfeiffer that adds libgap compilation and
+[might be merged](https://github.com/gap-system/gap/pull/1205)
+soon in the devel version of GAP.

-See https://github.com/markuspf/gap/issues/2 for the few sticking points that could prevent using a vanilla GAP from the distribution (please edit further if you think about more of them).
+See https://github.com/markuspf/gap/issues/2 for the few sticking points
+that could prevent using a vanilla GAP from the distribution (please edit
+further if you think about more of them).

 What the branch does:

@@ -159,3 +166,8 @@

This should be fixed once GAP's gap binary is built on top of libgap. See: https://github.com/markuspf/gap/issues/1. + +Note: + +- GAP 4.9.1 was released in May 2018.

antonio-rojas commented 6 years ago
comment:63

I'm confused - I just built GAP 4.9.1 from the upstream tarball and 'make install' does install /usr/lib/libgap.so and libgap's headers. Is this a different libgap than the one mentioned here to be included in 4.10?

slel commented 6 years ago
comment:64

One comment at pull request 1205 in GAP's github repo said GAP won't ship libGAP before GAP 4.10:

Maybe this was changed though.

nthiery commented 6 years ago

Description changed:

--- 
+++ 
@@ -171,3 +171,4 @@

 - [GAP 4.9.1 was released](https://mail.gap-system.org/pipermail/forum/2018/005741.html) in May 2018.
   This is the first stable release of GAP 4.9 (GAP 4.9.0 was a beta for GAP 4.9).
+- Max Horn reviewed the list of GAP symbols we use in Sage; some have already changed in 4.9. See this [pad](https://hackmd.io/emNi76svSWCh1fBeLKqPdA?edit#) for notes.
jdemeyer commented 6 years ago
comment:66

Has anybody asked GAP developers for a policy on the prefixing?

In Cernay, I proposed that GAP should prefix some/all of their names like T_INT -> GAP_T_INT. For compiling GAP (+ GAP packages) itself, we can use a file with defines like

#define  T_INT  GAP_T_INT

to ensure backwards compatibility. This way, in GAP itself, one could use either T_INT or GAP_T_INT. Outside of GAP, only GAP_T_INT would work.

I think this is relatively easy to implement (I'm willing to do it) but I don't know what upstream GAP thinks of it.

slel commented 6 years ago

Description changed:

--- 
+++ 
@@ -169,6 +169,7 @@

 Note:

-- [GAP 4.9.1 was released](https://mail.gap-system.org/pipermail/forum/2018/005741.html) in May 2018.
-  This is the first stable release of GAP 4.9 (GAP 4.9.0 was a beta for GAP 4.9).
+- [GAP 4.9.2](http://www.gap-system.org/Releases/4.9.2.html) was released in July 2018.
+- [GAP 4.9.1](http://www.gap-system.org/Releases/4.9.1.html), released in May 2018,
+  was the first stable release of GAP 4.9 (GAP 4.9.0 was a beta for GAP 4.9).
 - Max Horn reviewed the list of GAP symbols we use in Sage; some have already changed in 4.9. See this [pad](https://hackmd.io/emNi76svSWCh1fBeLKqPdA?edit#) for notes.
slel commented 6 years ago
comment:68

GAP 4.9.2 was released in July 2018.

nthiery commented 6 years ago
comment:70

Hi,

For the record: I experimented a bit a month ago and a bit more this morning with Dima with the latest gap master that has make libgap and make install-libgap rules. See the following branch, which builds libgap from GAP's pristine sources:

https://github.com/sagemath/sagetrac-mirror/commits/u/nthiery/gap-libgap

Note that, at this stage, Sage won't start with it; maybe even not compile.

nthiery commented 6 years ago
comment:71

Has anybody asked GAP developers for a policy on the prefixing?

In Cernay, I proposed that GAP should prefix some/all of their names like T_INT -> GAP_T_INT. For compiling GAP (+ GAP packages) itself, we can use a file with defines like

#define  T_INT  GAP_T_INT

to ensure backwards compatibility. This way, in GAP itself, one could use either T_INT or GAP_T_INT. Outside of GAP, only GAP_T_INT would work.

I think this is relatively easy to implement (I'm willing to do it) but I don't know what upstream GAP thinks of it.

That's the idea indeed. Eventually they will provide a standalone header file that will define the official GAP API, and where all symbols will be prefixed. To bootstrap the process, we (well mostly Max!) reviewed and annotated the symbols that are used in Sage:

https://hackmd.io/emNi76svSWCh1fBeLKqPdA#
timokau commented 6 years ago
comment:72

Thank you for working on this! Sage using an up-to-date gap and doing away with the additional libgap package will simplify packaging.

videlec commented 6 years ago
comment:73

update milestone 8.3 -> 8.4

slel commented 6 years ago
comment:74

GAP 4.9.3 was released.

dimpase commented 6 years ago

Changed work issues from Wait for gap 4.9 release to Wait for gap 4.10 release

dimpase commented 6 years ago

Description changed:

--- 
+++ 
@@ -2,7 +2,7 @@
 our packaging. In fact, it may well enable Sage to use a vanilla GAP
 installation as provided by the distribution.

-The GAP team will not provide libGAP with GAP 4.9, but only with GAP 4.10.
+The GAP team will provide libGAP with GAP 4.10.

 The branch attached to this ticket updates Sage to run on top of
 [a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
dimpase commented 6 years ago

Description changed:

--- 
+++ 
@@ -1,18 +1,14 @@
-GAP 4.9 comes with a completely rewritten build system that will simplify
+GAP 4.9 came with a completely rewritten build system that will simplify
 our packaging. In fact, it may well enable Sage to use a vanilla GAP
 installation as provided by the distribution.

 The GAP team will provide libGAP with GAP 4.10.

-The branch attached to this ticket updates Sage to run on top of
-[a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
-by Markus Pfeiffer that adds libgap compilation and
-[might be merged](https://github.com/gap-system/gap/pull/1205)
-soon in the devel version of GAP.
+The branch attached to this ticket updates Sage to run on top of the current GAP dev branch. (the  branch of GAP 
+by Markus Pfeiffer that adds libgap compilation has been merged).

-See https://github.com/markuspf/gap/issues/2 for the few sticking points
-that could prevent using a vanilla GAP from the distribution (please edit
-further if you think about more of them).
+Some historical notes may be found in https://github.com/markuspf/gap/issues/2
+

 What the branch does:

@@ -53,7 +49,7 @@
 can explain, e.g. change of orders in lists of elements. So those
 should be nothing to worry about. There is not much point in updating
 those doctests right away; we may as well wait for a more final
-version of 4.9 to be out.
+version of 4.10 to be out.

 TODO:

@@ -70,14 +66,11 @@

 - ???

-Fetching Markus's GAP sources:
+Fetching  GAP sources:

-git clone git@github.com:markuspf/gap.git $LIBGAP +git clone git@github.com:gap-system/gap.git $LIBGAP cd $LIBGAP -git remote add markuspf git@github.com:markuspf/gap.git -git fetch markuspf -git checkout -b markuspf/hpc-merge-libgap ./autogen.sh ./configure make bootstrap-pkg-minimal @@ -87,9 +80,9 @@ Testing libgap:

-./configure --enable-libgap
-make -j4 libgap
-make test-libgap
+./configure
+make -j4
+make testlibgap

Build and install a tardist for Sage, and rebuild the spkg: @@ -101,7 +94,7 @@ make manuals make clean

-(cd ..; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git $GAP) +(cd ..; mv $GAP src; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git src)

sage --package fix-checksum sage -f gap # -s

dimpase commented 6 years ago

Description changed:

--- 
+++ 
@@ -1,14 +1,18 @@
-GAP 4.9 came with a completely rewritten build system that will simplify
+GAP 4.9 comes with a completely rewritten build system that will simplify
 our packaging. In fact, it may well enable Sage to use a vanilla GAP
 installation as provided by the distribution.

 The GAP team will provide libGAP with GAP 4.10.

-The branch attached to this ticket updates Sage to run on top of the current GAP dev branch. (the  branch of GAP 
-by Markus Pfeiffer that adds libgap compilation has been merged).
+The branch attached to this ticket updates Sage to run on top of
+[a branch of GAP](https://github.com/markuspf/gap/tree/hpc-merge-libgap)
+by Markus Pfeiffer that adds libgap compilation and
+[might be merged](https://github.com/gap-system/gap/pull/1205)
+soon in the devel version of GAP.

-Some historical notes may be found in https://github.com/markuspf/gap/issues/2
-
+See https://github.com/markuspf/gap/issues/2 for the few sticking points
+that could prevent using a vanilla GAP from the distribution (please edit
+further if you think about more of them).

 What the branch does:

@@ -49,7 +53,7 @@
 can explain, e.g. change of orders in lists of elements. So those
 should be nothing to worry about. There is not much point in updating
 those doctests right away; we may as well wait for a more final
-version of 4.10 to be out.
+version of 4.9 to be out.

 TODO:

@@ -66,11 +70,14 @@

 - ???

-Fetching  GAP sources:
+Fetching Markus's GAP sources:

-git clone git@github.com:gap-system/gap.git $LIBGAP +git clone git@github.com:markuspf/gap.git $LIBGAP cd $LIBGAP +git remote add markuspf git@github.com:markuspf/gap.git +git fetch markuspf +git checkout -b markuspf/hpc-merge-libgap ./autogen.sh ./configure make bootstrap-pkg-minimal @@ -80,9 +87,9 @@ Testing libgap:

-./configure
-make -j4
-make testlibgap
+./configure --enable-libgap
+make -j4 libgap
+make test-libgap

Build and install a tardist for Sage, and rebuild the spkg: @@ -94,7 +101,7 @@ make manuals make clean

-(cd ..; mv $GAP src; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git src) +(cd ..; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git $GAP)

sage --package fix-checksum sage -f gap # -s

dimpase commented 6 years ago
comment:77

note that the prefix of the tarball should start with src/ - else one gets weird error messages from the python's tar.

GAPs make install target now works in a preliminary form - on Linux for sure.

dimpase commented 6 years ago

Changed branch from u/jdemeyer/upgrade_to_gap_4_9 to u/dimpase/WIP/libgap410

dimpase commented 6 years ago
comment:78

This is WIP - builds, but libgap.pyx interaction loop has to be rewritten using new API.

Also, sage --gap is broken

gap: hmm, I cannot find 'lib/init.g' maybe use option '-l <gaproot>'?

New commits:

c766ad6Merge branch 'u/jdemeyer/upgrade_to_gap_4_9' of trac.sagemath.org:sage into libgap
5152637building GAP and libGAP from GAP
6bed83achanges for using libGAP from GAP - part 1
f069122removing util*
dimpase commented 6 years ago

Changed commit from 40e644a to f069122

dimpase commented 6 years ago
comment:79

oops, one still needs initialize() from util.pyx, in some form...

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from f069122 to 9c22e72

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

8c28e13Revert "removing util*"
9c22e72few comments
nthiery commented 6 years ago
comment:81

Great to see the progress! Sounds like a productive week in Siegen. Thanks so much.

dimpase commented 6 years ago

Description changed:

--- 
+++ 
@@ -173,3 +173,5 @@
 - [GAP 4.9.1](http://www.gap-system.org/Releases/4.9.1.html), released in May 2018,
   was the first stable release of GAP 4.9 (GAP 4.9.0 was a beta for GAP 4.9).
 - Max Horn reviewed the list of GAP symbols we use in Sage; some have already changed in 4.9. See this [pad](https://hackmd.io/emNi76svSWCh1fBeLKqPdA?edit#) for notes.
+
+Tarball : https://github.com/dimpase/gap/releases/download/dev/gap-4.dev.tar.gz
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from 9c22e72 to 14919d3

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

14919d3update util and the GAP package
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from 14919d3 to 8b5ccbe

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

b3dc304Merge branch 'develop' into libgap
8a8a146use MarkBag instead of MARK_BAG
c5a5d4fno more *_BAG, and a call signature fix
d13fc80more compilation fixes
8b5ccberemoved debugging utils (they need a rewrite)
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from 8b5ccbe to a0616de

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a0616deadding --nointeract to libgap startup options
dimpase commented 6 years ago
comment:86

at this point I am at

sage: libgap(1)
Error, Variable: '_rich_repr_' must have a value
Error, Variable: '_ipython_canary_method_should_not_exist_' must have a\
 value
[... several more errors like above]
gap: panic, could not open *errout*!
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 6 years ago

Changed commit from a0616de to 9cc327f