Upgrade: GAP 4.12

[slel]

GAP 4.12.1 fixes critical bugs in 4.12.0

GAP 4.12.1 release page

GAP 4.12.0 was released in August 2022.

• GAP 4.12.0 release announcement
• GAP 4.12.0 release page
• GAP 4.12.0: changes since GAP 4.11.1

We should upgrade our corresponding spkgs:

• gap
• gap_packages
• libsemigroups

Previous upgrades:

• 31498: GAP 4.11.1

• 29314: GAP 4.11, libsemigroup 1.0.9

Upstream: Completely fixed; Fix reported upstream

CC: @antonio-rojas @dimpase @slel @tornaria @kiwifb @collares @tscrim

Component: packages: standard

Keywords: spkg, upgrade, gap, libsemigroups

Author: Antonio Rojas, Dima Pasechnik

Reviewer: Dima Pasechnik, Antonio Rojas, Mauricio Collares, François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/34391

[antonio-rojas]

comment:2

Having upgraded downstream, here is a list of issues:

1) Build fails because a garbage collector needs to be specified via a preprocessor macro when using gap's headers. This is included in the CPPFLAGS in gap's own sysinfo.gap, but Sage doesn't read it when building.

2) Build fails due to API changes in OpenOutputStream and CloseOutput

3) Colored prompt is on by default, which breaks the pexpect interface.

4) The NaturalHomomorphism function, used in groups.abelian_gps.abelian_group_gap, has been removed.

5) The functions IsPrimitive and IsTransitive changed behavior and now return false if the group does not act on the given set, breaking some tests:

File "/usr/lib/python3.10/site-packages/sage/groups/perm_gps/permgroup.py", line 4363, in sage.groups.perm_gps.permgroup.PermutationGroup_generic.is_transitive
Failed example:
    G.is_transitive([1,4,5])
Expected:
    True
Got:
    False
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/groups/perm_gps/permgroup.py", line 4428, in sage.groups.perm_gps.permgroup.PermutationGroup_generic.is_primitive
Failed example:
    G.is_primitive([1,2,3])
Expected:
    True
Got:
    False

6) Computation of permutation group generators has changed, breaking some tests. In particular, using algorithm=smaller now does not make any difference in many examples.

7) Several trivial test failures due to different choice of generators or minor syntax changes.

[antonio-rojas]

Branch: u/arojas/upgrade__gap_4_12

[antonio-rojas]

comment:4

The branch fixes (2), (3), (4) and (7). For (5), we need to decide whether we want to follow the new behavior in GAP, or reimplement the old behavior in some other way. For (6), some tests need to be updated or removed.

---

New commits:

aeff992	Adapt to API changes in OpenOutputStream and CloseOutput
c3367b4	Disable colored prompt as it breaks the pexpect interface
3b63e99	Port NaturalHomomorphism uses
bc40764	Fix tests with GAP 4.12

[antonio-rojas]

Commit: bc40764

[antonio-rojas]

Author: Antonio Rojas, ...

[dimpase]

comment:6

regarding (5), I don't think we need to reimplement, as transitive and primitive are dealing with group action on a domain, and are not defined, mathematically, for arbitrary subsets of the domain. (perhaps we should have a deprecation, in case False is returned, printing a warning about the change in behavior)

How about (1) - should spkg-install source sysinfo.gap ?

[kiwifb]

comment:7

Replying to @dimpase:

regarding (5), I don't think we need to reimplement, as transitive and primitive are dealing with group action on a domain, and are not defined, mathematically, for arbitrary subsets of the domain.

How about (1) - should spkg-install source sysinfo.gap ?

I think that is fine for gap packages but not for downstream packages. I guess it is a stop gap, but a standard way of setting the flags should be provided. A .pc file or at least a config utility. You shouldn't have to search for settings in an obscure internal file. This is a genuine upstream defect in my opinion.

[dimpase]

comment:8

By the way, sysinfo.gap is not in the tarball, it is generated by doing make sysinfo.gap, so this should be easy to incorporate. But it has GAP_CPPFLAGS rather than CPPFLAGS, at least on an install I'm trying this.

So this looks that our build somehow manages to avoid running make sysinfo.gap.

[collares]

comment:11

I am seeing a bunch of segfaults on aarch64 with GAP 4.12.0 and this branch. The following patch, which copies the output string before calling GAP_Leave, seems to fix almost all of the segfaults, but not 100% of them (see https://github.com/NixOS/nixpkgs/pull/192548#issuecomment-1256686607 for more details):

diff --git a/src/sage/libs/gap/element.pxd b/src/sage/libs/gap/element.pxd
index a1bf8118d4..9a28de87ca 100644
--- a/src/sage/libs/gap/element.pxd
+++ b/src/sage/libs/gap/element.pxd
@@ -29,9 +29,9 @@ cdef GapElement_Boolean make_GapElement_Boolean(parent, Obj obj)
 cdef GapElement_Function make_GapElement_Function(parent, Obj obj)
 cdef GapElement_Permutation make_GapElement_Permutation(parent, Obj obj)

-cdef char *capture_stdout(Obj, Obj)
-cdef char *gap_element_str(Obj)
-cdef char *gap_element_repr(Obj)
+cdef str capture_stdout(Obj, Obj)
+cdef str gap_element_str(Obj)
+cdef str gap_element_repr(Obj)

 cdef class GapElement(RingElement):
diff --git a/src/sage/libs/gap/element.pyx b/src/sage/libs/gap/element.pyx
index e2681165a2..482f74bbcf 100644
--- a/src/sage/libs/gap/element.pyx
+++ b/src/sage/libs/gap/element.pyx
@@ -120,7 +120,7 @@ cdef Obj make_gap_matrix(sage_list, gap_ring) except NULL:
     return l.value

-cdef char *capture_stdout(Obj func, Obj obj):
+cdef str capture_stdout(Obj func, Obj obj):
     """
     Call a single-argument GAP function ``func`` with the argument ``obj``
     and return the stdout from that function call.
@@ -152,12 +152,12 @@ cdef char *capture_stdout(Obj func, Obj obj):

         CALL_1ARGS(func, obj)
         CloseOutput(&output)
-        return CSTR_STRING(s)
+        return char_to_str(CSTR_STRING(s))
     finally:
         GAP_Leave()

-cdef char *gap_element_repr(Obj obj):
+cdef str gap_element_repr(Obj obj):
     """
     Implement ``repr()`` of ``GapElement``s using the ``ViewObj()`` function,
     which is by default closest to what you get when displaying an object in
@@ -169,7 +169,7 @@ cdef char *gap_element_repr(Obj obj):
     return capture_stdout(func, obj)

-cdef char *gap_element_str(Obj obj):
+cdef str gap_element_str(Obj obj):
     """
     Implement ``str()`` of ``GapElement``s using the ``Print()`` function.

@@ -761,7 +761,7 @@ cdef class GapElement(RingElement):
         if  self.value == NULL:
             return 'NULL'

-        s = char_to_str(gap_element_str(self.value))
+        s = gap_element_str(self.value)
         return s.strip()

     def _repr_(self):
@@ -783,7 +783,7 @@ cdef class GapElement(RingElement):
         if  self.value == NULL:
             return 'NULL'

-        s = char_to_str(gap_element_repr(self.value))
+        s = gap_element_repr(self.value)
         return s.strip()

     cpdef _set_compare_by_id(self):

[collares]

comment:12

I think the other segfaults happen because gap_element_repr and gap_element_str (the two capture_stdout callers) call GAP_ValueGlobalVariable but don't use GAP_Enter/GAP_Leave.

As a proof of concept, I applied the above patch but also moved the try/GAP_Enter/finally/GAP_Leave block from capture_stdout to gap_element_repr/gap_element_str (patch), and I didn't see a single crash in two test runs (it used to happen every time).

[antonio-rojas]

comment:13

Replying to Mauricio Collares:

I think the other segfaults happen because gap_element_repr and gap_element_str (the two capture_stdout callers) call GAP_ValueGlobalVariable but don't use GAP_Enter/GAP_Leave.

As a proof of concept, I applied the above patch but also moved the try/GAP_Enter/finally/GAP_Leave block from capture_stdout to gap_element_repr/gap_element_str (patch), and I didn't see a single crash in two test runs (it used to happen every time).

Feel free to push your changes, I can only test on x86_64 where I haven't seen any such issue so far.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

cbc902a	Adapt test to new is_transitive and is_primitive behavior
80dd6b3	Mark test as random. With gap 4.12 on x86_64 it is no longer 2
a6c293d	Remove test that is now redundant, all seeds give the same answer

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bc40764 to a6c293d

[tornaria]

comment:15

I have tested the last commmit (a6c293d) using system gap 4.12.0, on x86_64, x86_64-musl and i686 (void linux) with all long tests passing.

I don't have aarch64 to test.

[dimpase]

comment:16

this branch should bump up the version of GAP, etc

[dimpase]

comment:17

diff --git a/build/pkgs/gap/checksums.ini b/build/pkgs/gap/checksums.ini
index 066e943308..7f083710ca 100644
--- a/build/pkgs/gap/checksums.ini
+++ b/build/pkgs/gap/checksums.ini
@@ -1,5 +1,5 @@
 tarball=gap-VERSION.tar.gz
-sha1=4ecdd281b8f430282fb9b12690b06e0a26abde10
-md5=85dc9e459d5b6c66fcad9f468afd3e3e
-cksum=1351843158
+sha1=b8b2d803c43dc6ea4413b5a378689a2087c3658a
+md5=6772845916c31de880792c7bb1672f81
+cksum=3760719912
 upstream_url=https://github.com/gap-system/gap/releases/download/vVERSION/gap-VERSION.tar.gz
diff --git a/build/pkgs/gap/package-version.txt b/build/pkgs/gap/package-version.txt
index d782fca8f6..815588ef14 100644
--- a/build/pkgs/gap/package-version.txt
+++ b/build/pkgs/gap/package-version.txt
@@ -1 +1 @@
-4.11.1
+4.12.0

[antonio-rojas]

comment:18

We should wait for 4.12.1 which will include a fix for issue (1) and additional fixes for make install so we can switch spkg-install to use it.

[dimpase]

Changed commit from a6c293d to 09f4df2

[dimpase]

comment:19

meanwhile, rebased over the latest beta, and all the fixes for spkg-install needed so far.

---

New commits:

6f3b455	Adapt to API changes in OpenOutputStream and CloseOutput
31aaa12	Disable colored prompt as it breaks the pexpect interface
d9da790	Port NaturalHomomorphism uses
86a34a0	Fix tests with GAP 4.12
3b8283f	Adapt test to new is_transitive and is_primitive behavior
754b7b1	Mark test as random. With gap 4.12 on x86_64 it is no longer 2
6e3e8da	Remove test that is now redundant, all seeds give the same answer
09f4df2	install gap 4.12.0

[dimpase]

Changed branch from u/arojas/upgrade__gap_4_12 to u/dimpase/upgrade__gap_4_12

[dimpase]

comment:20

running tests on Apple M1 (aka aarch64).

[dimpase]

comment:21

I'm getting

    In file included from sage/coding/codecan/codecan.c:5227:
    /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/include/gap/gasman_intern.h:17:2: error: #error This file must only be included if GASMAN is used
       17 | #error This file must only be included if GASMAN is used

what's the fix here?

[antonio-rojas]

comment:22

That's precisely issue (1). Fixed upstream in https://github.com/gap-system/gap/commit/276eb56919400aab91f9e00abb94ddc41a0ffabc

[dimpase]

comment:23

OK, one needs to patch GAP, as in https://patch-diff.githubusercontent.com/raw/gap-system/gap/pull/5077.diff

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 09f4df2 to 5a0f6e9

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5a0f6e9

add GAP upstream PR 5077

[dimpase]

comment:25

we also need to upgrade libsemigroups, and gap_packages. I'm doing this now.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5cfe772	upgrade libsemigroup and gap_packages
61025c0	Merge branch 'u/dimpase/upgrade__gap_4_12' of trac.sagemath.org:sage into u/dimpase/upgrade__gap_4_12

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5a0f6e9 to 61025c0

[dimpase]

comment:27

not yet done, as local/share/gap/sysinfo.gap needed a manual addition of -I$SAGE_LOCAL/include/gap to GAC_CFLAGS.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

2e3f54b

also, fix the package names etc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 61025c0 to 2e3f54b

[dimpase]

comment:29

There are also failing doctests in sage/algebras/quantum_groups/quantum_group_gap.py, like this:

sage -t --warn-long 37.8 --random-seed=5244030565037623767917974460011040182 src/sage/algebras/quantum_groups/quantum_group_gap.py
**********************************************************************
File "src/sage/algebras/quantum_groups/quantum_group_gap.py", line 1812, in sage.algebras.quantum_groups.quantum_group_gap.TensorProductOfHighestWeightModules.__init__
Failed example:
    TestSuite(T).run()  # optional - gap_packages
Expected nothing
Got:
    doctest:warning
      File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/src/bin/sage-runtests", line 154, in <module>
        err = DC.run()
... 
    DeprecationWarning: implementations of Modules().TensorProducts() now must define the method tensor_factors

[dimpase]

Changed author from Antonio Rojas, ... to Antonio Rojas, Dima Pasechnik

[dimpase]

comment:30

Perhaps Travis can fix these missing methods tensor_factors in sage/algebras/quantum_groups/quantum_group_gap.py, so that there won't be a need to patch these doctests.

[mkoeppe]

comment:31

see #34635

[dimpase]

Work Issues: simplify spkg-install to use GAP make install to $prefix

[dimpase]

wip on simplifying spkg-install

[dimpase]

comment:33

Attachment: s.patch.gz

with this, and https://github.com/gap-system/gap/pull/5091 it mostly works (package installation is broken though) We'll wait for 4.12.1, where also needed package upgrades are done.

[tornaria]

comment:34

In case it's useful, here are the patches I had to use for gap 4.12.0 on void linux:

https://github.com/void-linux/void-packages/tree/ed7862591d1ff3a617b58ecbe2efdbb5343d508b/srcpkgs/gap/patches

I think two of those are PR 5077 and PR 5091 already mentioned in this ticket. The third patch I needed to fix something in atlasrep package.

Note also that: a. package names have changed (at least the name of the directories changed case and removed version numbers) b. sysinfo.gap used to be installed in /usr/share/gap and now is installed in /usr/lib/gap.

Sagemath (9.7 and 9.8.beta2) is working fine with this, but note I only install the minimal set of packages that are standard in sage.

[collares]

comment:35

The atlasrep issue is https://github.com/gap-system/gap/issues/5015 and will also be fixed in GAP 4.12.1.

[dimpase]

comment:36

Replying to Gonzalo Tornaría:

In case it's useful, here are the patches I had to use for gap 4.12.0 on void linux:

https://github.com/void-linux/void-packages/tree/ed7862591d1ff3a617b58ecbe2efdbb5343d508b/srcpkgs/gap/patches

I think two of those are PR 5077 and PR 5091 already mentioned in this ticket. The third patch I needed to fix something in atlasrep package.

Note also that: a. package names have changed (at least the name of the directories changed case and removed version numbers) b. sysinfo.gap used to be installed in /usr/share/gap and now is installed in /usr/lib/gap.

For Sage's installed GAP, this will be $SAGE_LOCAL/lib/gap/

Sagemath (9.7 and 9.8.beta2) is working fine with this, but note I only install the minimal set of packages that are standard in sage.

Try packages which build executables and/or GAP kernel extensions (the latter need gac) E.g. grape and crypting.

[dimpase]

Changed commit from 2e3f54b to d0a9b21

[dimpase]

Changed branch from u/dimpase/upgrade__gap_4_12 to u/dimpase/packages/gap/4_12_1

[dimpase]

comment:37

update tp 4.12.1

---

New commits:

d503bc8	add GAP upstream PR 5077
894a89f	also, fix the package names etc
d0a9b21	update GAP to 4.12.1

[dimpase]

comment:38

GAP package liepring now depends on a GAP package singular, so we should be installing it ahead of liepring.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

02b6b4b

install GAP package singular - new dependency of liepring

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d0a9b21 to 02b6b4b

[dimpase]

comment:40

while more simplifications may be done on spkg-install files for gap, and gap_packages, this branch appears to work just fine.

[antonio-rojas]

comment:41

utils and autodoc are now dependencies of atlasrep so they need to move from gap_packages to gap.