search

racket / pkg-build

Other
5 stars 9 forks source link

don't try to stop crashed container #10

Closed michaelballantyne closed 4 years ago

michaelballantyne commented 4 years ago

Unlike a VM, a docker container crashes when out of memory. I would expect the build to record a failure and move on to the next package, but currently the build crashes:

raco test: 1 "/home/root//user/.racket/snapshot/pkgs/cur-test/cur/tests/ntac/software-foundations/Induction-ML.rkt"                                                                           
Induction-ML.rkt: subprocess: process creation failed                                                                                                                                         
Racket virtual machine has run out of memory; aborting                                                                                                                                        
user break                                                                                                                                                                                    
Timeout after 2400 seconds                                                                                                                                                                    
*** test failed ***                                                                                                                                                                           
fatal error: runtime: out of memory                                                                                                                                                           

runtime stack:                                                                                                                                                                                
runtime.throw(0x55eefbfc8a5c, 0x16)                                                                                                                                                           
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/panic.go:617 +0x74 fp=0x7fffb143f4b0 sp=0x7fffb143f480 pc=0x55eefa9ec5d4                                                     
runtime.sysMap(0xc000000000, 0x4000000, 0x55eefdf8afb8)                                                                                                                                       
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mem_linux.go:170 +0xc9 fp=0x7fffb143f4f0 sp=0x7fffb143f4b0 pc=0x55eefa9d78e9                                                 
runtime.(*mheap).sysAlloc(0x55eefdf71aa0, 0x2000, 0x55eefdf71ab0, 0x1)                                                                                                                        
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:633 +0x1cf fp=0x7fffb143f598 sp=0x7fffb143f4f0 pc=0x55eefa9ca6ff                                                   
runtime.(*mheap).grow(0x55eefdf71aa0, 0x1, 0x0)                                                                                                                                               
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1222 +0x44 fp=0x7fffb143f5f0 sp=0x7fffb143f598 pc=0x55eefa9e4cf4
runtime.(*mheap).allocSpanLocked(0x55eefdf71aa0, 0x1, 0x55eefdf8afc8, 0x0)                                                                                                                    
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1150 +0x381 fp=0x7fffb143f628 sp=0x7fffb143f5f0 pc=0x55eefa9e4be1                                                   
runtime.(*mheap).alloc_m(0x55eefdf71aa0, 0x1, 0x2a, 0x6e43a318)                                                                                                                               
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:977 +0xc6 fp=0x7fffb143f678 sp=0x7fffb143f628 pc=0x55eefa9e4236                                                     
runtime.(*mheap).alloc.func1()                                                                                                                                                                
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1048 +0x4e fp=0x7fffb143f6b0 sp=0x7fffb143f678 pc=0x55eefaa1535e                                                    
runtime.(*mheap).alloc(0x55eefdf71aa0, 0x1, 0x55eefa01002a, 0x7fffb143f750)                                                                                                                   
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1047 +0x8c fp=0x7fffb143f700 sp=0x7fffb143f6b0 pc=0x55eefa9e450c                                                    
runtime.(*mcentral).grow(0x55eefdf728a0, 0x0)                                                                                                                                                 
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcentral.go:256 +0x97 fp=0x7fffb143f748 sp=0x7fffb143f700 pc=0x55eefa9d7367                                                  
runtime.(*mcentral).cacheSpan(0x55eefdf728a0, 0x7f0034d5b000)                                                                                                                                 
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcentral.go:106 +0x301 fp=0x7fffb143f7a8 sp=0x7fffb143f748 pc=0x55eefa9d6e71                                                 
runtime.(*mcache).refill(0x7f0034d5b008, 0x2a)                                                                                                                                                
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcache.go:135 +0x88 fp=0x7fffb143f7c8 sp=0x7fffb143f7a8 pc=0x55eefa9d6908                                                    
runtime.(*mcache).nextFree(0x7f0034d5b008, 0x55eefdf6792a, 0x7f0034d5b008, 0x7f0034d5b000, 0x8)                                                                                               
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:786 +0x8a fp=0x7fffb143f800 sp=0x7fffb143f7c8 pc=0x55eefa9caf3a                                                    
runtime.mallocgc(0x180, 0x55eefccbbe20, 0x1, 0x55eefdf8b020)                                                                                                                                  
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:939 +0x780 fp=0x7fffb143f8a0 sp=0x7fffb143f800 pc=0x55eefa9cb870                                                   
runtime.newobject(0x55eefccbbe20, 0x4000)                                                                                                                                                     
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:1068 +0x3a fp=0x7fffb143f8d0 sp=0x7fffb143f8a0 pc=0x55eefa9cbc7a                                                   
runtime.malg(0x1fb5c00008000, 0x55eefdf74110)                                                                                                                                                 
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:3220 +0x33 fp=0x7fffb143f910 sp=0x7fffb143f8d0 pc=0x55eefa9f5a83
runtime.mpreinit(...)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/os_linux.go:311
runtime.mcommoninit(0x55eefdf6bd60)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:618 +0xc6 fp=0x7fffb143f948 sp=0x7fffb143f910 pc=0x55eefa9ef3f6
runtime.schedinit()
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:540 +0x78 fp=0x7fffb143f9a0 sp=0x7fffb143f948 pc=0x55eefa9ef088
runtime.rt0_go(0x7fffb143faa8, 0x4, 0x7fffb143faa8, 0x0, 0x7f003438eb97, 0x4, 0x7fffb143faa8, 0x400008000, 0x55eefaa173d0, 0x0, ...)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/asm_amd64.s:195 +0x11e fp=0x7fffb143f9a8 sp=0x7fffb143f9a0 pc=0x55eefaa174fe
docker-stop: stop failed
  name: "pkg-build"
  context...:
   ...ow-val-first.rkt:486:18
   /home/ubuntu/pkg-build/main.rkt:870:2: build-pkg-set
   [repeats 1 more time]
   /home/ubuntu/pkg-build/main.rkt:93:0: build-pkgs
   (submod "/home/ubuntu/build.rkt" main): [running body]
   temp35_0
   for-loop
   run-module-instance!
ubuntu@ip-172-31-3-220:~$

I believe the crash was due to the call to vm-stop, which calls docker-stop, which throws if the container is not running.

This PR modifies vm-stop to be a no-op if the container isn't running. I'm not sure whether this is the correct solution---we want to be sure to record a failure. At least in the specific case I ran into, the package was recorded in test-fail via a timeout failure, so the existing logic may be enough.

mflatt commented 4 years ago

Thanks!