Location of binaries in Windows distribution

[olexandr-konovalov]

When GAP for WIndows is built under Cygwin, binaries are placed in .libs directory. After that the GAP root directory is archived and copied to a Cygwin-free machine for testing. But bat-files in the bin directory are referring to bin\i686-pc-cygwin-gcc-default32\gap.exe which is not there - it is in .libs instead.

A workaround which works is to move the whole content of .libs into the bin/i686-pc-cygwin-gcc-default32 directory. I am now doing this when testing GAP distribution on Jenkins. But what would be the proper way to do this?

[fingolfin]

GAP binaries on all architectures are not anymore in bin/GAPARCH, and hence also not on Windows. Instead, they are in the GAP root directory.

So I am not sure if you are saying that it isn't there, either?

Anyway, I see references to \bin\@GAPARCH resp. /bin/@GAPARCH@ in files in cnf/cygwin. Those should be adjusted to instead reference the GAP base directory.

[olexandr-konovalov]

@fingolfin ah, indeed there is gap.exe in the GAP root directory too, but of different size:

gapw7-build+jenkins@gapw7-build /cygdrive/c/gap-4.9.0
$ ls
aclocal.m4      configure        gap.ini         libgap.la           README.md
autogen.sh      configure.ac     gen             libtool             src
autom4te.cache  CONTRIBUTING.md  GNUmakefile     LICENSE             sysinfo.gap
bin             doc              GNUmakefile.in  Makefile            sysinfo.gap-default32
CITATION        etc              grp             Makefile.rules      terminfo
cnf             extern           hpcgap          obj                 tst
config.log      gac              INSTALL.md      pkg
config.status   gap.exe          lib             README.buildsys.md

gapw7-build+jenkins@gapw7-build /cygdrive/c/gap-4.9.0
$ ls .libs/
cyggap-0.dll  gap.exe  gap_ltshwrapper  libgap.dll.a  libgap.la  libgap.lai  lt-gap.c

gapw7-build+jenkins@gapw7-build /cygdrive/c/gap-4.9.0
$ ls -la gap.exe
-rwxrwxr-x+ 1 gapw7-build+jenkins gapw7-build+None 29198 Dec  3 08:44 gap.exe

gapw7-build+jenkins@gapw7-build /cygdrive/c/gap-4.9.0
$ ls -la .libs/gap.exe
-rwxrwxr-x+ 1 gapw7-build+jenkins gapw7-build+None 62730 Dec  3 08:44 .libs/gap.exe

I will see if pointing bat file to that will work instead of copying files from .libs.

[ChrisJefferson]

We need to look over how the chain executable is built. We apply some "magic" to it (run a command on it to increase the heap), and we should check which are being built in which ways. Having two different executables isn't good -- let's figure out which one we want.

[fingolfin]

Actually, most likely one of the two isn't a real exe, but rather a shell script invoking the other one (at least that it's how it would be on a UNIX system in a comparable situation). @alex-konovalov perhaps the file tool can test this for you, i.e. file gap.exe .libs/gap.exe

The reason for this is, very roughly speaking, to deal with DLL hell in an uninstalled executable. Not that we really care about, we actually kind of rely on it when bundling our own copy of the cygwin DLL (sigh).

[olexandr-konovalov]

@fingolfin here it is:

$ file gap.exe .libs/gap.exe
gap.exe:       PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows
.libs/gap.exe: PE32 executable (console) Intel 80386, for MS Windows

[fingolfin]

Ah OK, then it's not as bad -- no shell script here, just a stripped executable. So just use the regular one, not .lib/gap.exe

[olexandr-konovalov]

I am having now another look at windows, downloading the latest gap-4.9.0-win.zip release candidate built on December 3rd. Trying to start it with bin/gap.bat.

1) The path in gap.bat has the line with ${DISTNAME}

start "GAP" C:\${DISTNAME}\bin\i686-pc-cygwin-gcc-default32\mintty.exe -s 120,40 /proc/cygdrive/C/gap-4.9.0/bin/i686-pc-cygwin-gcc-default32/gap.exe -l /proc/cygdrive/C/gap-4.9.0 %*

so the first error is that mintty.exe is not found. Fixing that by replacing ${DISTNAME} by gap-4.9.0.

2) the path to gap.exe is incorrect - removing bin/i686-pc-cygwin-gcc-default32/ from it.

3) above that line, fix another ${DISTNAME} appearance to

set PATH=C:\gap-4.9.0\bin\i686-pc-cygwin-gcc-default32;%PATH% 

4) Now GAP starts:

 ┌───────┐   GAP 4.9.0 of 03-Dec-2017
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: i686-pc-cygwin-gcc-default32
 Configuration:  gmp 6.0.0, readline
 Loading the library and packages ...
 Packages:   AClib 1.2, Alnuth 3.1.0, AtlasRep 1.5.1, AutPGrp 1.8, CRISP 1.4.4, Cryst 4.1.12, CrystCat 1.1.6,
             CTblLib 1.2.2, FactInt 1.5.4, FGA 1.3.1, GAPDoc 1.6.1, IO 4.5.0, IRREDSOL 1.4, LAGUNA 3.8.0,
             Polenta 1.3.8, Polycyclic 2.11, PrimGrp 3.2.0, RadiRoot 2.7, ResClasses 4.6.0, smallgrp 1.2, Sophus 1.23,
             SpinSym 1.5, TomLib 1.2.6, TransGrp 2.0.2, utils 0.48
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap>

So indeed, there is no need to copy the content of .libs into bin/i686-pc-cygwin-gcc-default32. I will trace how bat files are handled by the build system and release wrapping tools to fix them.

[olexandr-konovalov]

Also had a look at the ability to load packages. LoadAllPackages() is broken because of the following:

@fingolfin - ANUPQ

The system cannot find the path specified.
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `+' on 2 arguments
The 1st argument is 'fail' which might point to an earlier problem
 at /proc/cygdrive/C/gap-4.9.0/lib/methsel2.g:250 called from
<function "HANDLE_METHOD_NOT_FOUND">( <arguments> )
 called from read-eval loop at /proc/cygdrive/C/gap-4.9.0/pkg/anupq-3.2/lib/anupqhead.gi:40
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> 

@fingolfin - ACE

The system cannot find the path specified.
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `+' on 2 arguments
The 1st argument is 'fail' which might point to an earlier problem
 at /proc/cygdrive/C/gap-4.9.0/lib/methsel2.g:250 called from
<function "HANDLE_METHOD_NOT_FOUND">( <arguments> )
 called from read-eval loop at /proc/cygdrive/C/gap-4.9.0/pkg/ace-5.2/gap/ace.gi:47
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk>

@james-d-mitchell - Digraphs

#W dlopen() error: No such file or directory
Error, module '/proc/cygdrive/C/gap-4.9.0/pkg/digraphs-0.11.0/bin/i686-pc-cygwin-gcc-default32/digraphs.so' not found in
  if not LOAD_DYN( arg[1], false ) then
    Error( "no support for dynamic loading" );
fi; at /proc/cygdrive/C/gap-4.9.0/lib/files.gd:583 called from
<function "LoadDynamicModule">( <arguments> )
 called from read-eval loop at /proc/cygdrive/C/gap-4.9.0/pkg/digraphs-0.11.0/init.g:25
Error, was not in any namespace at /proc/cygdrive/C/gap-4.9.0/lib/variable.g:291 called from
LEAVE_NAMESPACE(  ); at /proc/cygdrive/C/gap-4.9.0/lib/package.gi:1210 called from
ReadPackage( pkgname, "init.g" ); at /proc/cygdrive/C/gap-4.9.0/lib/package.gi:1583 called from
LoadPackage( n[i] ); at *stdin*:7 called from
<function "unknown">( <arguments> )
 called from read-eval loop at *stdin*:7
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk>

@ChrisJefferson - JSON

#W dlopen() error: No such file or directory
Error, module '/proc/cygdrive/C/gap-4.9.0/pkg/json-1.2.0/bin/i686-pc-cygwin-gcc-default32/json.so' not found in
  if not LOAD_DYN( arg[1], false ) then
    Error( "no support for dynamic loading" );
fi; at /proc/cygdrive/C/gap-4.9.0/lib/files.gd:583 called from
<function "LoadDynamicModule">( <arguments> )
 called from read-eval loop at /proc/cygdrive/C/gap-4.9.0/pkg/json-1.2.0/init.g:8
Error, was not in any namespace at /proc/cygdrive/C/gap-4.9.0/lib/variable.g:291 called from
LEAVE_NAMESPACE(  ); at /proc/cygdrive/C/gap-4.9.0/lib/package.gi:1210 called from
ReadPackage( pkgname, "init.g" ); at /proc/cygdrive/C/gap-4.9.0/lib/package.gi:1583 called from
func( C[i] ) at /proc/cygdrive/C/gap-4.9.0/lib/coll.gi:746 called from
List( RecNames( GAPInfo.PackagesInfo ), LoadPackage ); at /proc/cygdrive/C/gap-4.9.0/lib/package.gi:1626 called from
<function "LoadAllPackages">( <arguments> )
 called from read-eval loop at *stdin*:10
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk>

For these packages, calling LoadPackage again after exiting from the break loop returns true, although they are clearly broken.

The list of packages which cleanly fail to load, some by legitimate reasons, is:

[ "help", "browse", "normalizinterface", "polymakeinterface", "zeromqinterface", 
"carat", "float", "itc",  "linboxing", "pargap", "profiling", "semigroups", "xgap" ]

[ChrisJefferson]

Have these ever worked in a released cygwin GAP?

[olexandr-konovalov]

I have an impression that JSON worked before at some point, but can't provide more exact information at the moment. Anyhow, even if they did not work, their loading should cleanly fail, instead of an error. To compare with GAP 4.8.8 installation:

1) ANUPQ fails to load cleanly:

gap> LoadPackage("anupq");
#I  ANUPQ package is not available. To see further details, enter
#I  SetInfoLevel(InfoPackageLoading,4); and try to load the package again.
fail

2) Same error for ACE:

gap> LoadPackage("ace");
The system cannot find the path specified.
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `+' on 2 arguments called from
<function "HANDLE_METHOD_NOT_FOUND">( <arguments> )
 called from read-eval loop at line 47 of /proc/cygdrive/C/gap4r8p8/pkg/ace-5.2/gap/ace.gi
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue

3) Same error for JSON:

gap> LoadPackage("json");
#W dlopen() error: No such file or directory
Error, module '/proc/cygdrive/C/gap4r8p8/pkg/json-1.1.0/bin/i686-pc-cygwin-gcc-default32/json.so' not found in
  if not LOAD_DYN( arg[1], false )  then
    Error( "no support for dynamic loading" );
fi; at /proc/cygdrive/C/gap4r8p8/lib/files.gd:579 called from
<function "LoadDynamicModule">( <arguments> )
 called from read-eval loop at line 8 of /proc/cygdrive/C/gap4r8p8/pkg/json-1.1.0/init.g
Error, was not in any namespace called from
LEAVE_NAMESPACE(  ); at /proc/cygdrive/C/gap4r8p8/lib/package.gi:1212 called from
ReadPackage( pkgname, "init.g" ); at /proc/cygdrive/C/gap4r8p8/lib/package.gi:1585 called from
<function "LoadPackage">( <arguments> )
 called from read-eval loop at line 2 of *stdin*
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> 

4) Same for Digraphs:

gap> LoadPackage("digraphs");
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Loading  orb 4.7.6 (Methods to enumerate orbits)
by Juergen Mueller (http://www.math.rwth-aachen.de/~Juergen.Mueller),
   Max Neunhöffer (http://www-groups.mcs.st-and.ac.uk/~neunhoef), and
   Felix Noeske (http://www.math.rwth-aachen.de/~Felix.Noeske).
Homepage: https://gap-packages.github.io/orb
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Loading  GRAPE 4.7 (GRaph Algorithms using PErmutation groups)
by Leonard H. Soicher (http://www.maths.qmul.ac.uk/~leonard/).
Homepage: http://www.maths.qmul.ac.uk/~leonard/grape/
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#W dlopen() error: No such file or directory
Error, module '/proc/cygdrive/C/gap4r8p8/pkg/digraphs-0.10.1/bin/i686-pc-cygwin-gcc-default32/digraphs.so' not found in
  if not LOAD_DYN( arg[1], false )  then
    Error( "no support for dynamic loading" );
fi; at /proc/cygdrive/C/gap4r8p8/lib/files.gd:579 called from
<function "LoadDynamicModule">( <arguments> )
 called from read-eval loop at line 25 of /proc/cygdrive/C/gap4r8p8/pkg/digraphs-0.10.1/init.g
Error, was not in any namespace called from
LEAVE_NAMESPACE(  ); at /proc/cygdrive/C/gap4r8p8/lib/package.gi:1212 called from
ReadPackage( pkgname, "init.g" ); at /proc/cygdrive/C/gap4r8p8/lib/package.gi:1585 called from
<function "LoadPackage">( <arguments> )
 called from read-eval loop at line 1 of *stdin*
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk>

Digraphs issue reported at https://github.com/gap-packages/Digraphs/issues/86

[olexandr-konovalov]

@ChrisJefferson @james-d-mitchell @fingolfin: to compare, this happens when packages are compiled under Cygwin and GAP started with bin/gap.sh in Cygwin shell:

gap> LoadPackage("anupq");
The system cannot find the path specified.
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `+' on 2 arguments
The 1st argument is 'fail' which might point to an earlier problem
 at /cygdrive/c/Users/jenkins/workspace/GAP-master-windows-build/GAP-master-sn\
apshot/lib/methsel2.g:250 called from
<function "HANDLE_METHOD_NOT_FOUND">( <arguments> )
 called from read-eval loop at /cygdrive/c/Users/jenkins/workspace/GAP-master-\
windows-build/GAP-master-snapshot/pkg/anupq-3.2/lib/anupqhead.gi:40
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> quit;
gap> LoadPackage("ace");
The system cannot find the path specified.
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `+' on 2 arguments
The 1st argument is 'fail' which might point to an earlier problem
 at /cygdrive/c/Users/jenkins/workspace/GAP-master-windows-build/GAP-master-sn\
apshot/lib/methsel2.g:250 called from
<function "HANDLE_METHOD_NOT_FOUND">( <arguments> )
 called from read-eval loop at /cygdrive/c/Users/jenkins/workspace/GAP-master-\
windows-build/GAP-master-snapshot/pkg/ace-5.2/gap/ace.gi:47
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> quit;
gap> LoadPackage("digraphs");
─────────────────────────────────────────────────────────────────────────────
Loading  orb 4.8.0 (Methods to enumerate orbits)
by Juergen Mueller (http://www.math.rwth-aachen.de/~Juergen.Mueller),
   Max Neunhöffer (http://www-groups.mcs.st-and.ac.uk/~neunhoef), and
   Felix Noeske (http://www.math.rwth-aachen.de/~Felix.Noeske).
Homepage: https://gap-packages.github.io/orb
─────────────────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────────────────
Loading  GRAPE 4.7 (GRaph Algorithms using PErmutation groups)
by Leonard H. Soicher (http://www.maths.qmul.ac.uk/~leonard/).
Homepage: http://www.maths.qmul.ac.uk/~leonard/grape/
─────────────────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────────────────
Loading  Digraphs 0.11.0 (Digraphs - Methods for digraphs)
by J. De Beule (http://homepages.vub.ac.be/~jdbeule/),
   J. Jonusas (http://www-groups.mcs.st-andrews.ac.uk/~julius/),
   J. D. Mitchell (http://goo.gl/ZtViV6),
   M. Torpey (http://www-groups.mcs.st-andrews.ac.uk/~mct25/), and
   W. A. Wilson (http://www-groups.mcs.st-andrews.ac.uk/~waw7/).
Homepage: https://gap-packages.github.io/Digraphs
─────────────────────────────────────────────────────────────────────────────
true
gap> LoadPackage("json");
─────────────────────────────────────────────────────────────────────────────
Loading  json 1.2.0 (Reading and Writing JSON)
by Christopher Jefferson (http://caj.host.cs.st-andrews.ac.uk/).
Homepage: https://gap-packages.github.io/json/
─────────────────────────────────────────────────────────────────────────────
true
gap>

[fingolfin]

@alex-konovalov So, isn't the original issue resolved? And all your recent comments are about issues with specific packages -- so I'd imagine each should be a separate issue, on the respective package issue tracker, no? (And also tracked under https://github.com/gap-system/gap/projects/4?)

[fingolfin]

BTW, it's strange that the listed path like /proc/cygdrive/C/gap-4.9.0/pkg/json-1.2.0/bin/i686-pc-cygwin-gcc-default32/json.so contain a GAP arch which contains "gcc", i.e. the compiler name -- in current master, this should be gone.

Perhaps check the full content of /proc/cygdrive/C/gap-4.9.0/pkg/json-1.2.0/bin/ (and also for the other packages -- all errors indicate that the packages failed to load some binary).

[olexandr-konovalov]

@fingolfin original issue on the way to resolving - let me first adjust references to \bin\@GAPARCH resp. /bin/@GAPARCH@ in files in cnf/cygwin, and check that all release pipeline works, and convert reports about packages into separate issues.

[olexandr-konovalov]

The reason was in missing cygstdc++-6.dll, as suggested by @ChrisJefferson. I have now added this to the -win.zip archive, and both Digraphs and JSON now can be loaded on Windows. So for these two packages the problem will disappear in the next GAP distribution.

[fingolfin]

Then I guess this can be closed (of course anupq and ACE are unchanged, but (a) these issues should be reported on their issue trackers, not here, and (b) even if, I have no idea how to fix them, as I now very little about cygwin, and have no good way to test them (only via a clunky VM on an external HD I have to dig out of a closet)

[olexandr-konovalov]

Problems with ANUPQ and ACE have been reported at https://github.com/gap-packages/anupq/issues/1 and https://github.com/gap-packages/ace/issues/13. For now, in the GAP distribution for Windows, I will remove anupq-*/bin and ace-*/bin directory to ensure that LoadAllPackages(); works, but it will be nice to fix this in the long run to ensure that LoadAllPackages(); does not result in a break loop on Windows without any provisional arrangements.

[ChrisJefferson]

I refer to this in #2023. I think it's GAP's fault, rather than the packages.

[olexandr-konovalov]

Seems so - both ace and anupq now loadable under Windows in GAP 4.11.1.