Something is making the builds in Travis CI fail, sometimes

[madworx]

Add debugging to expect and qemu to figure out what's really happening under the hood. I cannot reproduce this on local builds.

[madworx]

https://travis-ci.org/madworx/docker-minix/builds/406228365#L881

Just failed with an interesting case:

size on /dev/imgrd set to 0kB
Multiuser startup in progress ...
Starting hotplugging infrastructure... done.
Starting services: random e1000 inete1000#0: Int...
 pty uds ipc log printer.
Starting daemons: update cron syslogd.
Starting networking: dhcpd nonamed.
Local packages (start):  done.
Minix/i386 (10.0.2.15) (console)
login: 

^^^ Here expect should have put root, but instead just waited (until set timeout expired?)

---> 083322d4cb4c
Removing intermediate container c85497d52ade

... and as a result killed the container and broke the file system:

Step 17/29 : COPY tools/install-minix-stage3.expect /
 ---> bfb82b69c8e7
Step 18/29 : RUN /install-minix-stage3.expect
 ---> Running in 1dd9abcee5cc
spawn qemu-system-x86_64 -d int,pcall,cpu_reset,unimp,guest_errors -D /tmp/qemu.log -nographic -serial mon:stdio -drive format=qcow2,file=/minix.qcow2 -boot c
cSeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)
iPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F915A0+07EF15A0 C980

MINIX 3.3.0. (588a35b)
Copyright 2014, Vrije Universiteit, Amsterdam, The Netherlands
MINIX is open source software, see http://www.minix3.org
Started VFS: 9 worker thread(s)
Root device name is /dev/c0d0p0s0
/dev/c0d0p0s0: dirty, performing fsck
Checking zone map. Checking inode map. Checking inode list. 
207 files, 31 directories, 31981 free inodes, 23796 free zones
----- FILE SYSTEM MARKED CLEAN -----
/dev/c0d0p0s0 is mounted on /
none is mounted on /proc
/dev/c0d0p0s2: dirty, performing fsck
Checking zone map. zone 115353 is missing
zone 115355 is missing
zone 115356 is missing
zone 115357 is missing
zone 115358 is missing
zone 115359 is missing
zone 115360 is missing
zone 115361 is missing
zone 115362 is missing
zone 115363 is missing
zone 115364 is missing
zone 115365 is missing
zone 115366 is missing
13 errors found. install a new map? (y=yes, n=no, q=quit, A=for yes to all) yes
Checking inode map. Checking inode list. 
4751 files, 567 directories, 1287712 free inodes, 2472688 free zones
----- FILE SYSTEM HAS BEEN MODIFIED -----
----- FILE SYSTEM MARKED CLEAN -----

It's apparent that (a) expect isn't propagating the timeout condition via its program exit code, and (b) either an issue with the hardware, qemu or the minix kernel/os.

[madworx]

Expect scripts now exit with an error code if timeouts occur. (Implemented as part of #5) The issues hasn't happened again -- my guess would be a slow builder VM, combined with timeouts we didn't notice, making the expected output mismatch against the actuals.

Will leave this issue open until that theory can be considered confirmed.

[madworx]

https://travis-ci.org/madworx/docker-minix/builds/406294411

Latest build (merge against master) seem to confirm that the timeouts are too tight in some cases:

####### FAILED due to timeout.
Dumping last rows of expect debug log:
------------8<------------8<------------8<------------8<------------8<------------8<------------
expect: does "\u001bc\u001b[?7l\u001b[2J\u001b[0mSeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)\r\r\n\r\n\r\niPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F915A0+07EF15A0 C980\r\r\nPress Ctrl-B to configure iPXE (PCI 00:03.0)...\r                                                                               \r\r\n\r\n\r\nBooting from Hard Disk...\r\r\nNetBSD MBR boot\r\r\n\r\nMINIX/x86 minixfs3 Primary Bootstrap\r\r\n\u001b[01;01H\u001b[13;28H\r\r\n\u001b[14;24H\r\r\n\u001b[15;40H\u001b[16;40H\r\r\n\u001b[18;02H\r\r\n\u001b[19;41H\u001b[20;44H\u001b[20;45H\u001b[20;44H\u001b[21;41H\r\r\n\r\n" (spawn_id exp4) match glob pattern "login: "? no
expect: does "\u001bc\u001b[?7l\u001b[2J\u001b[0mSeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)\r\r\n\r\n\r\niPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F915A0+07EF15A0 C980\r\r\nPress Ctrl-B to configure iPXE (PCI 00:03.0)...\r                                                                               \r\r\n\r\n\r\nBooting from Hard Disk...\r\r\nNetBSD MBR boot\r\r\n\r\nMINIX/x86 minixfs3 Primary Bootstrap\r\r\n\u001b[01;01H\u001b[13;28H\r\r\n\u001b[14;24H\r\r\n\u001b[15;40H\u001b[16;40H\r\r\n\u001b[18;02H\r\r\n\u001b[19;41H\u001b[20;44H\u001b[20;45H\u001b[20;44H\u001b[21;41H\r\r\n\r\n\u001b[23;40H" (spawn_id exp4) match glob pattern "login: "? no
expect: does "\u001bc\u001b[?7l\u001b[2J\u001b[0mSeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)\r\r\n\r\n\r\niPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F915A0+07EF15A0 C980\r\r\nPress Ctrl-B to configure iPXE (PCI 00:03.0)...\r                                                                               \r\r\n\r\n\r\nBooting from Hard Disk...\r\r\nNetBSD MBR boot\r\r\n\r\nMINIX/x86 minixfs3 Primary Bootstrap\r\r\n\u001b[01;01H\u001b[13;28H\r\r\n\u001b[14;24H\r\r\n\u001b[15;40H\u001b[16;40H\r\r\n\u001b[18;02H\r\r\n\u001b[19;41H\u001b[20;44H\u001b[20;45H\u001b[20;44H\u001b[21;41H\r\r\n\r\n\u001b[23;40H\u001b[24;41H" (spawn_id exp4) match glob pattern "login: "? no
expect: does "\u001bc\u001b[?7l\u001b[2J\u001b[0mSeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)\r\r\n\r\n\r\niPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F915A0+07EF15A0 C980\r\r\nPress Ctrl-B to configure iPXE (PCI 00:03.0)...\r                                                                               \r\r\n\r\n\r\nBooting from Hard Disk...\r\r\nNetBSD MBR boot\r\r\n\r\nMINIX/x86 minixfs3 Primary Bootstrap\r\r\n\u001b[01;01H\u001b[13;28H\r\r\n\u001b[14;24H\r\r\n\u001b[15;40H\u001b[16;40H\r\r\n\u001b[18;02H\r\r\n\u001b[19;41H\u001b[20;44H\u001b[20;45H\u001b[20;44H\u001b[21;41H\r\r\n\r\n\u001b[23;40H\u001b[24;41H\u001b[25;42H" (spawn_id exp4) match glob pattern "login: "? no
expect: timed out
------------8<------------8<------------8<------------8<------------8<------------8<------------

[madworx]

Closing as we now have stable builds again.