search

CounterHack / HolidayHack2017

SANS Holiday Hack Challenge 2017
1 stars 0 forks source link

Snow Place Like Home Terminal #9

Closed misuchiru03 closed 6 years ago

misuchiru03 commented 6 years ago

Upon entering a query, the terminal crashes and displays runtime/cgo: pthread_create failed: resource temporarily unavailable.

Doesn't seem to be enough resources to be able to perform the steps to solve the challenge. The same goes for anything with a large dataset, such as Bumbles Bounce - Least Popular Browser.

Here is a snippit from the error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x7fb8e0c36fcf m=0 sigcode=18446744073709551610 goroutine 0 [idle]: goroutine 1 [syscall]: runtime.notetsleepg(0x1683ee8, 0xffffffffffffffff, 0xc42001a000) /usr/local/go/src/runtime/lock_futex.go:205 +0x42 fp=0xc420196fc8 sp=0xc420196f98 runtime.gcBgMarkStartWorkers() /usr/local/go/src/runtime/mgc.go:1413 +0xb1 fp=0xc420197000 sp=0xc420196fc8 runtime.gcStart(0x0, 0x100) /usr/local/go/src/runtime/mgc.go:997 +0x339 fp=0xc420197038 sp=0xc420197000 runtime.mallocgc(0x120, 0xee12e0, 0x1, 0x545a42) /usr/local/go/src/runtime/malloc.go:785 +0x491 fp=0xc4201970d8 sp=0xc420197038 runtime.newobject(0xee12e0, 0x0) /usr/local/go/src/runtime/malloc.go:820 +0x38 fp=0xc420197108 sp=0xc4201970d8 regexp.progMachine(0xc4201c70b0, 0xc4201c7110, 0x0) /usr/local/go/src/regexp/exec.go:73 +0x3f fp=0xc4201971d0 sp=0xc420197108 regexp.(Regexp).get(0xc4201e7c20, 0x0) /usr/local/go/src/regexp/regexp.go:216 +0xd4 fp=0xc420197210 sp=0xc4201971d0 regexp.(Regexp).doExecute(0xc4201e7c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffce8653f02, 0xb, 0x0, 0x0, ...) /usr/local/go/src/regexp/exec.go:418 +0x40 fp=0xc4201972b0 sp=0xc420197210 regexp.(Regexp).doMatch(0xc4201e7c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffce8653f02, 0xb, 0xc42001c600) /usr/local/go/src/regexp/exec.go:410 +0xc8 fp=0xc420197340 sp=0xc4201972b0 regexp.(Regexp).MatchString(0xc4201e7c20, 0x7ffce8653f02, 0xb, 0x16883c0) /usr/local/go/src/regexp/regexp.go:416 +0x6c fp=0xc420197398 sp=0xc420197340 github.com/docker/cli/vendor/github.com/docker/distribution/reference.ParseAnyReference(0x7ffce8653f02, 0xb, 0xc420444320, 0x0, 0x0, 0xe4d800) /go/src/github.com/docker/cli/vendor/github.com/docker/distribution/reference/normalize.go:145 +0x48 fp=0xc420197410 sp=0xc420197398 github.com/docker/cli/cli/command/container.createContainer(0x7fb8e15020c8, 0xc4204421c0, 0x163b5a0, 0xc4201f55c0, 0xc420444300, 0x7ffce8653edd, 0x24, 0x0, 0x0, 0x0) /go/src/github.com/docker/cli/cli/command/container/create.go:175 +0x13c fp=0xc4201975c0 sp=0xc420197410 github.com/docker/cli/cli/command/container.runContainer(0xc4201f55c0, 0xc42038ff20, 0xc420384f00, 0xc420444300, 0x0, 0x0) /go/src/github.com/docker/cli/cli/command/container/run.go:164 +0x228 fp=0xc420197ae0 sp=0xc4201975c0 github.com/docker/cli/cli/command/container.runRun(0xc4201f55c0, 0xc42009d4d0, 0xc42038ff20, 0xc420384f00, 0x0, 0x0) /go/src/github.com/docker/cli/cli/command/container/run.go:117 +0x39b fp=0xc420197c48 sp=0xc420197ae0 github.com/docker/cli/cli/command/container.NewRunCommand.func1(0xc4203b9d40, 0xc4201be780, 0x1, 0xf, 0x0, 0x0) /go/src/github.com/docker/cli/cli/command/container/run.go:49 +0x109 fp=0xc420197ca0 sp=0xc420197c48 github.com/docker/cli/vendor/github.com/spf13/cobra.(Command).execute(0xc4203b9d40, 0xc420010140, 0xf, 0x10, 0xc4203b9d40, 0xc420010140) /go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:646 +0x44e fp=0xc420197d58 sp=0xc420197ca0 github.com/docker/cli/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc420386480, 0xc420386900, 0xc420355a80, 0xc42000e4a0) /go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:742 +0x349 fp=0xc420197ea8 sp=0xc420197d58 github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420386480, 0xc420386480, 0x1629800) /go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:695 +0x2b fp=0xc420197ed8 sp=0xc420197ea8 main.main() /go/src/github.com/docker/cli/cmd/docker/docker.go:168 +0xc3 fp=0xc420197f88 sp=0xc420197ed8 runtime.main() /usr/local/go/src/runtime/proc.go:185 +0x20a fp=0xc420197fe0 sp=0xc420197f88 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420197fe8 sp=0xc420197fe0 /usr/local/go/src/bufio/bufio.go:97 +0x117

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 5 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:116 +0x104 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.1 /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 8 [IO wait]: net.runtime_pollWait(0x7fb8e15530e0, 0x72, 0x3) /usr/local/go/src/runtime/netpoll.go:164 +0x59 net.(pollDesc).wait(0xc4203f5bf8, 0x72, 0x162cf80, 0x16253f0) /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38 net.(pollDesc).waitRead(0xc4203f5bf8, 0xc420440000, 0x1000) /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34 net.(netFD).Read(0xc4203f5b90, 0xc420440000, 0x1000, 0x1000, 0x0, 0x162cf80, 0x16253f0) /usr/local/go/src/net/fd_unix.go:250 +0x1b7 net.(conn).Read(0xc42000e600, 0xc420440000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:181 +0x70 net/http.(persistConn).Read(0xc4203ef680, 0xc420440000, 0x1000, 0x1000, 0x4, 0x452d60, 0xc420428a20) /usr/local/go/src/net/http/transport.go:1316 +0x14b bufio.(Reader).fill(0xc4204288a0) /usr/local/go/src/bufio/bufio.go:97 +0x117 bufio.(Reader).Peek(0xc4204288a0, 0x1, 0xc42005dbe5, 0x1, 0x0, 0xc420428900, 0x0) /usr/local/go/src/bufio/bufio.go:129 +0x67 net/http.(persistConn).readLoop(0xc4203ef680) /usr/local/go/src/net/http/transport.go:1474 +0x196 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 9 [select]: net/http.(persistConn).writeLoop(0xc4203ef680) /usr/local/go/src/net/http/transport.go:1704 +0x43a created by net/http.(Transport).dialConn /usr/local/go/src/net/http/transport.go:1118 +0xa5a

rax 0x0 rbx 0x6 rcx 0x7fb8e0c36fcf rdx 0x0 rdi 0x2 rsi 0x7ffce86533d0 rbp 0xfd2913 rsp 0x7ffce8653448 r8 0x0 r9 0x7ffce86533d0 r10 0x8 r11 0x246 r12 0x2be1050 r13 0x34 r14 0x0 r15 0xf3 rip 0x7fb8e0c36fcf rflags 0x246 cs 0x33 fs 0x0 gs 0x0

dpendolino commented 6 years ago

@misuchiru03 thanks for the detailed bug report!

We're actually spinning up so many containers that even though resources like ram and cpu are available, other more esoteric limits are being hit. We actually ran out of stack space which the Docker user ( non-root ) can allocate, AWESOME!

I'm throwing another server into play and increasing those limits so we should be in a better spot going forward.