Tests don't seem to pass in 32-bit cygwin

[wilfwilson]

I've been tentatively looking into why Digraphs doesn't work properly in cygwin, at least in 32-bit (seems to be okay in 64-bit now). I'm using AppVeyor to experiment with a 32-bit cygwin setup for GAP and Digraphs. A lot of the problems with Digraphs seem to arise when we want to do things with reading from/writing to files, so I thought I would check how the io package behaves on its own. You can see a build of what happens here and the full log is here: LOG: io-tests-in-cygwin-32bit.txt (the build dies after 1 hour without completing).

I'll copy the most relevant parts of the test failures. In tst/bugfix.tst, it seems like IO_WriteLine is always off by one; and in tst/testgap.tst it looks like something is really wrong; I don't understand it. Perhaps I'm doing something wrong with my setup, but I thought it was worth pointing this out anyway. This uses the GAP master branch.

 ┌───────┐   GAP 4.dev of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: i686-pc-cygwin-default32-kv6
 Configuration:  gmp 6.1.2, GASMAN
 Loading the library and packages ...
 Packages:   GAPDoc 1.6.1, PrimGrp 3.3.1, SmallGrp 1.3, TransGrp 2.0.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> ─────────────────────────────────────────────────────────────────────────────
Loading  IO 4.5.4 (Bindings for low level C library I/O routines)
by Max Neunhöffer (http://www-groups.mcs.st-and.ac.uk/~neunhoef).
maintained by:
   Max Horn (http://www.quendi.de/math).
Homepage: https://gap-packages.github.io/io
Report issues at https://github.com/gap-packages/io/issues
─────────────────────────────────────────────────────────────────────────────
true
Architecture: i686-pc-cygwin-default32-kv6

testing: /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:4
# Input is:
IO_WriteLine(f, s{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:6
# Input is:
IO_WriteLine(f, s);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:8
# Input is:
IO_WriteLine(f, s{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:10
# Input is:
IO_WriteLine(f, s{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:12
# Input is:
IO_WriteLine(f, s2{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:14
# Input is:
IO_WriteLine(f, s2);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:16
# Input is:
IO_WriteLine(f, s2{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:18
# Input is:
IO_WriteLine(f, s2{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:32
# Input is:
IO_WriteLine(f, s{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:34
# Input is:
IO_WriteLine(f, s);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:36
# Input is:
IO_WriteLine(f, s{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:38
# Input is:
IO_WriteLine(f, s{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
     281 ms (0 ms GC) and 13.1MB allocated for bugfix.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/children.tst
    1985 ms (0 ms GC) and 2.48MB allocated for children.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/sendstringbackground.tst
     703 ms (0 ms GC) and 41.3KB allocated for sendstringbackground.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst
      1 [main] gap 1492 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
  32646 [main] gap 1492 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
  37305 [main] gap 1492 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
      3 [main] gap 2400 fork: child -1 - forked process 1492 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2912 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   1145 [main] gap 2912 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1409 [main] gap 2912 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 217235 [main] gap 2400 fork: child -1 - forked process 2912 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 684 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    417 [main] gap 684 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
    824 [main] gap 684 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 433463 [main] gap 2400 fork: child -1 - forked process 684 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 3024 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   6509 [main] gap 3024 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
  73122 [main] gap 3024 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 717433 [main] gap 2400 fork: child -1 - forked process 3024 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 1908 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    628 [main] gap 1908 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1973 [main] gap 1908 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 941757 [main] gap 2400 fork: child -1 - forked process 1908 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2856 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    584 [main] gap 2856 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1622 [main] gap 2856 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1201718 [main] gap 2400 fork: child -1 - forked process 2856 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 552 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   2350 [main] gap 552 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   3357 [main] gap 552 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1406641 [main] gap 2400 fork: child -1 - forked process 552 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2192 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   1050 [main] gap 2192 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   2687 [main] gap 2192 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1617796 [main] gap 2400 fork: child -1 - forked process 2192 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2296 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    520 [main] gap 2296 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1476 [main] gap 2296 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1726817 [main] gap 2400 fork: child -1 - forked process 2296 died unexpectedly, retry 0, exit code 0x100, errno 11
      2 [main] gap 2316 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    692 [main] gap 2316 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1562 [main] gap 2316 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1829935 [main] gap 2400 fork: child -1 - forked process 2316 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2244 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    488 [main] gap 2244 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1415 [main] gap 2244 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1949767 [main] gap 2400 fork: child -1 - forked process 2244 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 1760 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    593 [main] gap 1760 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1487 [main] gap 1760 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2054953 [main] gap 2400 fork: child -1 - forked process 1760 died unexpectedly, retry 0, exit code 0x100, errno 11
      2 [main] gap 1412 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    669 [main] gap 1412 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1702 [main] gap 1412 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2169496 [main] gap 2400 fork: child -1 - forked process 1412 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2824 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    588 [main] gap 2824 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1510 [main] gap 2824 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2270756 [main] gap 2400 fork: child -1 - forked process 2824 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 420 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    569 [main] gap 420 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1096 [main] gap 420 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2384865 [main] gap 2400 fork: child -1 - forked process 420 died unexpectedly, retry 0, exit code 0x100, errno 11
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:8
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, AppendList: <list2> must be a small list (not the value 'fail')
Error, Unable to read compressed file correctly: 6
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:9
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, Usage: IO_HasData( f ) with IsFile(f)
Error, Unable to read compressed file correctly: 6
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:10
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, Usage: IO_HasData( f ) with IsFile(f)
Error, Unable to read compressed file correctly: 6
########

...and then lots more errors like this.

[fingolfin]

Sorry, but I can't really support cygwin, I have no way to test there. PRs are welcome, though.

[wilfwilson]

Thanks for the reply @fingolfin. Beyond playing with AppVeyor (which is not practical) I think the Digraphs and Semigroups developers also don't have a way to test on cygwin (and no knowledge about how to develop for it, either). So perhaps we should either explicitly not support 32-bit cygwin too, or at least see about disabling the functionality in Digraphs that requires IO, when using 32-bit cygwin. (Everything seems to be fine with 64-bit for now). @james-d-mitchell

[olexandr-konovalov]

Just to add that I will consider not supporting 32-bit cygwin by the IO package quite unfortunate - e.g. SCSCP package works under Windows because IO works. I will be happy to help with testing.

[fingolfin]

@wilfwilson I tried to fix this in 2f86e9d7671b6be47ccbe5e40f8cf0b5f3625eac but can't test it.

Of course it'd be nice if somebody could set up AppVeyor for this repository, or try to make Travis Windows support working (which I never tried)

[wilfwilson]

Thanks @fingolfin, seems like a nice solution, it got rid of some of the failures in tst/bugfix.tst, but it didn't get everything (from AppVeyor, running IO master tst/testall.g in GAP master):

Running IO tests...
 ┌───────┐   GAP 4.dev of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: i686-pc-cygwin-default32-kv7
 Configuration:  gmp 6.1.2, GASMAN
 Loading the library and packages ...
 Packages:   GAPDoc 1.6.1, PrimGrp 3.3.1, SmallGrp 1.3, TransGrp 2.0.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> ─────────────────────────────────────────────────────────────────────────────
Loading  IO 4.6.0dev (Bindings for low level C library I/O routines)
by Max Neunhöffer (http://www-groups.mcs.st-and.ac.uk/~neunhoef).
maintained by:
   Max Horn (https://www.quendi.de/math).
Homepage: https://gap-packages.github.io/io
Report issues at https://github.com/gap-packages/io/issues
─────────────────────────────────────────────────────────────────────────────
Architecture: i686-pc-cygwin-default32-kv7

testing: /home/appveyor/gap/pkg/io/tst/bugfix.tst
########> Diff in /home/appveyor/gap/pkg/io/tst/bugfix.tst:43
# Input is:
IO_WriteNonBlocking(f, s2, 1, 2^17) - cygwinAdjust;
# Expected output:
131072
# But found:
131071
########
########> Diff in /home/appveyor/gap/pkg/io/tst/bugfix.tst:45
# Input is:
IO_WriteNonBlocking(f, s2, 1, 2^17+1) - cygwinAdjust;
# Expected output:
131073
# But found:
131072
########
     156 ms (0 ms GC) and 13.1MB allocated for bugfix.tst
testing: /home/appveyor/gap/pkg/io/tst/children.tst

children.tst has been running for something like 10 minutes so far.

UPDATE: The tests stopped running! Unsurprisingly (since you didn't touch them), the other test files that failed before still fail.

[wilfwilson]

I feel that 32-bit GAP (at least on Windows) maybe isn't long for this world, what with OSes dropping 32-bit support, so as far as I'm concerned this isn't a priority.

[fingolfin]

Ah, my "fix" was a bit overeager. If you could try again... ?

Well, what I don't quite get is why it would work in 64 bit Windows?! Because those changes I made are windows specific, not 32 bit windows specific.

[wilfwilson]

You'll be able to see the result here https://ci.appveyor.com/project/wilfwilson/digraphs/builds/25937245. I wasn't looking at the 64-bit bit stuff yesterday, that also seems to have failed. Let's see what happens now.

[wilfwilson]

bugfix.tst passes, thanks: https://ci.appveyor.com/project/wilfwilson/digraphs/builds/25937355/job/abbwwyykjqs1p4od

testgap and subsequent files don't, seems to be related to forking. (Also, some of the files take ages to run.)

[fingolfin]

Yes, forking is super slow, esp. on 64bit windows, because it has to be emulated in a complicated way. See also https://github.com/gap-system/gap/pull/3513.

It's one of the various reasons why I am not a big fan of the idea of "just making io a required package and let's call that our official interface to everything". I really hope that one day, we'll progress beyond POSIX based operating system to something more sane, and wouldn't want to tie GAP closely to that.