search

uber / zanzibar

A build system & configuration system to generate versioned API gateways.
MIT License
465 stars 67 forks source link

Random hang of test suite #209

Open Raynos opened 7 years ago

Raynos commented 7 years ago

Sometimes the ./test/ package itself will hang in a test case

Here is a dump of sigabort to see why the tests / goroutines are hanging


{"level":"info","ts":1505849366.9648693,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4765,"zone":"unknown","realHTTPAddr":"10.32.106.72:34521","realTChannelAddr":"10.32.106.72:40161","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:38385","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:40114","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"error","ts":1505849367.029912,"msg":"Failed to Bootstrap in TestStartGateway()","error":"error listening on port: listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"error","ts":1505849367.0298743,"msg":"Error listening on port","hostname":"raynos-ThinkPad-T440p","pid":4776,"zone":"unknown","error":"listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"info","ts":1505849367.0834262,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","realHTTPAddr":"10.32.106.72:36721","realTChannelAddr":"10.32.106.72:37823","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:33303","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49837","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.0845563,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:33878","pathname":"/health","host":"10.32.106.72:36721","timestamp-started":1505849367.0845108,"timestamp-finished":1505849367.0845234,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.126711,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","realHTTPAddr":"10.32.106.72:35879","realTChannelAddr":"10.32.106.72:35419","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:40493","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:51699","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.1276844,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:37640","pathname":"/health","host":"10.32.106.72:35879","timestamp-started":1505849367.127651,"timestamp-finished":1505849367.1276588,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.1945999,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4802,"zone":"unknown","realHTTPAddr":"10.32.106.72:39537","realTChannelAddr":"10.32.106.72:44285","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:57802","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49706","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":true,"metrics.runtime.enableGCMetrics":true,"metrics.runtime.enableMemMetrics":true,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
fatal error: concurrent map iteration and map write

goroutine 50 [running]:
runtime.throw(0xba8457, 0x26)
    /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc42004d988 sp=0xc42004d968
runtime.mapiternext(0xc420443f80)
    /usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc42004da38 sp=0xc42004d988
reflect.mapiternext(0xc420443f80)
    /usr/local/go/src/runtime/hashmap.go:1156 +0x2b fp=0xc42004da50 sp=0xc42004da38
reflect.Value.MapKeys(0xac5220, 0xc420332b10, 0x15, 0xc42004dbe9, 0x98, 0xc42004dc38)
    /usr/local/go/src/reflect/value.go:1112 +0x1bc fp=0xc42004db00 sp=0xc42004da50
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.includeElement(0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc420040000)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:617 +0x2db fp=0xc42004dbc8 sp=0xc42004db00
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.Contains(0xf3cdc0, 0xc420338340, 0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc42004dd98, 0x2, 0x2, 0x1)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:645 +0x59 fp=0xc42004dc48 sp=0xc42004dbc8
github.com/uber/zanzibar/test_test.TestRuntimeMetrics(0xc420338340)
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/health_test.go:223 +0x731 fp=0xc42004dfa8 sp=0xc42004dc48
testing.tRunner(0xc420338340, 0xbb66a8)
    /usr/local/go/src/testing/testing.go:657 +0x96 fp=0xc42004dfd0 sp=0xc42004dfa8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004dfd8 sp=0xc42004dfd0
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420071520, 0xb98dc4, 0x12, 0xbb66a8, 0xc4202d5d01)
    /usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420071520)
    /usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420071520, 0xc4202d5de0)
    /usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420299d40, 0xf723e0, 0x4, 0x4, 0xaf4c20)
    /usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc420049f20, 0xc4202d5f20)
    /usr/local/go/src/testing/testing.go:822 +0xfc
main.main()
    github.com/uber/zanzibar/test/_test/_testmain.go:50 +0xf7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 37 [IO wait]:
net.runtime_pollWait(0x7f7d951b1948, 0x72, 0xd)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446298, 0x72, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446298, 0xc42048c028, 0xfde8)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420446230, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4203ca0a8, 0xc42048c028, 0xfde8, 0xfde8, 0xc42048bfc8, 0x411598, 0xc42048bfc8)
    /usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc420394480, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420388300, 0xc4204332d0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 52 [IO wait]:
net.runtime_pollWait(0x7f7d951b1708, 0x72, 0xf3eb00)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c458, 0x72, 0xf386e0, 0xc420421800)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c458, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42042c3f0, 0x0, 0xf3ca80, 0xc420421800)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4204c8088, 0xc420373a40, 0xabc420, 0xf6ade0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4204c8088, 0xc420373a10, 0xabc420, 0xf6ade0, 0xb23780)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204e00b0, 0xf43180, 0xc4204c8088, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4204cca20)
    /usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
    /usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7f7d951b1e88, 0x72, 0xf3eb00)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa78, 0x72, 0xf386e0, 0xc42033c000)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa78, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202daa10, 0x0, 0xf3ca80, 0xc42033c000)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e148, 0xc4203320c0, 0xabc420, 0xf6ade0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e148, 0xc420332090, 0xabc420, 0xf6ade0, 0xb23780)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001ec60, 0xf43180, 0xc42000e148, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc42006fce0)
    /usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
    /usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f7d951b1f48, 0x72, 0x5)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa08, 0x72, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa08, 0xc420312028, 0xfde8)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202da9a0, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e138, 0xc420312028, 0xfde8, 0xfde8, 0xc420311fc8, 0x411598, 0xc420311fc8)
    /usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2480, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420019d80, 0xc4202aed80)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 36 [IO wait]:
net.runtime_pollWait(0x7f7d951b1888, 0x72, 0xf3eb00)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446308, 0x72, 0xf386e0, 0xc42033c0c0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446308, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204462a0, 0x0, 0xf3ca80, 0xc42033c0c0)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4203ca0b8, 0xc420332330, 0xabc420, 0xf6ade0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4203ca0b8, 0xc420332300, 0xabc420, 0xf6ade0, 0xb23780)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204480b0, 0xf43180, 0xc4203ca0b8, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc420442180)
    /usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
    /usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 53 [IO wait]:
net.runtime_pollWait(0x7f7d951b17c8, 0x72, 0xf)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c3e8, 0x72, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c3e8, 0xc42051e028, 0xfde8)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42042c380, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4204c8078, 0xc42051e028, 0xfde8, 0xfde8, 0x411f6d, 0xc4204d0500, 0xc42051dfc8)
    /usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc42033a630, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fe380, 0xc420337a40)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f7d951b1c48, 0x72, 0xf3eb00)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db1e8, 0x72, 0xf386e0, 0xc420396000)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db1e8, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db180, 0x0, 0xf3ca80, 0xc420396000)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e270, 0xc42038c0c0, 0xabc420, 0xf6ade0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e270, 0xc42038c090, 0xabc420, 0xf6ade0, 0xb23780)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001efd0, 0xf43180, 0xc42000e270, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203441e0)
    /usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
    /usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 13 [IO wait]:
net.runtime_pollWait(0x7f7d951b1d08, 0x72, 0x9)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db178, 0x72, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db178, 0xc4203b8028, 0xfde8)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db110, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e260, 0xc4203b8028, 0xfde8, 0xfde8, 0xc4203b7fc8, 0x411598, 0xc4203b7fc8)
    /usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2870, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc180, 0xc420360110)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 51 [runnable]:
bytes.(*Buffer).WriteByte(0xc420110f50, 0xc42037592b, 0x1c, 0x1c)
    /usr/local/go/src/bytes/buffer.go:236 +0x82
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.KeyForPrefixedStringMap(0xc420375900, 0x1c, 0xc420038ca0, 0xc420038dd8, 0xc420038ca0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/key_gen.go:68 +0x267
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Service).EmitMetricBatch(0xc420300550, 0xc4203755e0, 0x0, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:159 +0x30b
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*m3ProcessorEmitMetricBatch).Process(0xc420337960, 0xc400000006, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0xfa00, 0x525, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:163 +0xeb
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*M3Processor).Process(0xc42033d7c0, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0x0, 0x0, 0xc4203755c0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:137 +0x306
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Server).Serve(0xc4203005a0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:95 +0x269
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).setupMetrics
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway.go:127 +0xc5

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f7d951b1ac8, 0x72, 0xf3eb00)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db728, 0x72, 0xf386e0, 0xc42033c020)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db728, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db6c0, 0x0, 0xf3ca80, 0xc42033c020)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e390, 0xc4203321b0, 0xabc420, 0xf6ade0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e390, 0xc420332180, 0xabc420, 0xf6ade0, 0xb23780)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001f340, 0xf43180, 0xc42000e390, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203444e0)
    /usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
    /usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 18 [IO wait]:
net.runtime_pollWait(0x7f7d951b1b88, 0x72, 0x8)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db6b8, 0x72, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db6b8, 0xc4203fe028, 0xfde8)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db650, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e380, 0xc4203fe028, 0xfde8, 0xfde8, 0xc4203fdfc8, 0x411598, 0xc4203fdfc8)
    /usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2d80, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc680, 0xc4203619c0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 23 [select]:
net/http.(*persistConn).readLoop(0xc42045aea0)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 21 [select]:
net/http.(*persistConn).readLoop(0xc420424000)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 22 [select]:
net/http.(*persistConn).writeLoop(0xc420424000)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc42045aea0)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 43 [syscall]:
syscall.Syscall(0x0, 0x11, 0xc4204e6000, 0x1000, 0x20, 0x20, 0xaf6700)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0x7f7d951f5960, 0x0, 0xc420027cf0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
syscall.Read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0xaf6700, 0x736901, 0xc42000c560)
    /usr/local/go/src/syscall/syscall_unix.go:162 +0x49
os.(*File).read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0xc, 0xc, 0x0)
    /usr/local/go/src/os/file_unix.go:165 +0x4f
os.(*File).Read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0x0, 0x0, 0x8)
    /usr/local/go/src/os/file.go:101 +0x76
bufio.(*Reader).fill(0xc420442060)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).ReadSlice(0xc420442060, 0x73c70a, 0xc4203ca008, 0xf4bd60, 0xc420094000, 0x0, 0xb331c0)
    /usr/local/go/src/bufio/bufio.go:338 +0xbb
bufio.(*Reader).ReadBytes(0xc420442060, 0xf4bd0a, 0xc420094000, 0xf4bd60, 0xc420094000, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:416 +0x66
bufio.(*Reader).ReadString(0xc420442060, 0xc42001010a, 0xc4200140e0, 0xc420027fc0, 0xc420094000, 0xc42052e000)
    /usr/local/go/src/bufio/bufio.go:456 +0x38
github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).copyToStdout(0xc4204ec000, 0xc420442060)
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:221 +0x34
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).createAndSpawnChild
    /home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:125 +0x61f
SIGABRT: abort
PC=0x45c2d3 m=2 sigcode=0

goroutine 0 [idle]:
runtime.futex(0xf23f58, 0x0, 0x7f111938adc8, 0x0, 0x7f1100000000, 0x45bf66, 0x3c, 0x0, 0x7f111938ae10, 0x40f720, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:426 +0x23
runtime.futexsleep(0xf23f58, 0x0, 0xdf8475800)
    /usr/local/go/src/runtime/os_linux.go:62 +0xd7
runtime.notetsleep_internal(0xf23f58, 0xdf8475800, 0x0)
    /usr/local/go/src/runtime/lock_futex.go:174 +0xd0
runtime.notetsleep(0xf23f58, 0xdf8475800, 0x104363f71d3d01)
    /usr/local/go/src/runtime/lock_futex.go:194 +0x56
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3805 +0x135
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1179 +0x11e
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1149 +0x64

goroutine 1 [chan receive, 49 minutes]:
testing.(*T).Run(0xc420063380, 0xb5b1dc, 0x10, 0xb78860, 0xc4202bdcf0)
    /usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420063380)
    /usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420063380, 0xc4202bddb0)
    /usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420287b20, 0xf15000, 0x1, 0x1, 0xf24600)
    /usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc42004ff20, 0x1e)
    /usr/local/go/src/testing/testing.go:822 +0xfc
command-line-arguments.TestMain(0xc4202bdf20)
    /home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:44 +0x58
main.main()
    command-line-arguments/_test/_testmain.go:40 +0xf7

goroutine 17 [syscall, 49 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [syscall, 49 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 7 [select, 49 minutes, locked to thread]:
runtime.gopark(0xb7a250, 0x0, 0xb532ec, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc42002cf50, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc42002cf50)
    /usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 8 [chan receive, 49 minutes]:
command-line-arguments.listenOnSignals.func1(0xc420061740)
    /home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:57 +0x53
created by command-line-arguments.listenOnSignals
    /home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:62 +0xbc

goroutine 9 [semacquire, 49 minutes]:
sync.runtime_Semacquire(0xc4202c2a6c)
    /usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4202c2a60)
    /usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/uber/zanzibar/runtime.(*Gateway).Wait(0xc420089400)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:263 +0x2f
command-line-arguments.logAndWait(0xc420089400)
    /home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main.go:102 +0x2f0
command-line-arguments.TestStartGateway(0xc420063450)
    /home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:92 +0x461
testing.tRunner(0xc420063450, 0xb78860)
    /usr/local/go/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 10 [chan receive]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.(*reporter).process(0xc4202cac80)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:483 +0xe3
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.NewReporter
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:241 +0x811

goroutine 11 [select]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.(*scope).reportLoop(0xc420089540, 0x989680)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:241 +0x158
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally.newRootScope
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:180 +0x6b2

goroutine 12 [runnable]:
github.com/uber/zanzibar/runtime.(*runtimeCollector).Start.func1(0xc42001d680)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:184 +0x13f
created by github.com/uber/zanzibar/runtime.(*runtimeCollector).Start
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:192 +0x84

goroutine 13 [select]:
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc4203cb440)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:231 +0x324
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.NewRemoteReporter
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:201 +0x288

goroutine 20 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4e00, 0x72, 0xee8660)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a78, 0x72, 0xee2660, 0xc420716d60)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a78, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a10, 0x0, 0xee6720, 0xc420716d60)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e078, 0xc42070d1a0, 0xc42003ad90, 0x5dbb5d)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e078, 0x6f1463, 0xc42003ada8, 0xc42003ada0)
    /usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e078, 0xc42070d170, 0xa86f20, 0xf13fa0, 0xaeb820)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b0000, 0xeece20, 0xc42000e078, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474500, 0xc4202c2a60)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:164 +0x5cc

goroutine 21 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4ec0, 0x72, 0xee8660)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a08, 0x72, 0xee2660, 0xc420716d80)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a08, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204989a0, 0x0, 0xee6720, 0xc420716d80)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e070, 0xc42070d290, 0xc42003ed90, 0x5dbb5d)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e070, 0x6f1463, 0xc42003eda8, 0xc42003eda0)
    /usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e070, 0xc42070d260, 0xa86f20, 0xf13fa0, 0xaeb820)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b00b0, 0xeece20, 0xc42000e070, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474550, 0xc4202c2a60)
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
    /home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:168 +0xce9

goroutine 22 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4d40, 0x72, 0xee8660)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498ae8, 0x72, 0xee2660, 0xc420716760)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498ae8, 0xffffffffffffffff, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a80, 0x0, 0xee6720, 0xc420716760)
    /usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e080, 0xc400000008, 0xc420480000, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e080, 0xb796e8, 0xc420716560, 0x1, 0x0)
    /usr/local/go/src/net/tcpsock.go:228 +0x49
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet.(*listener).Accept(0xc420716560, 0x0, 0x0, 0x0, 0x0)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet/listener.go:80 +0x8f
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).serve(0xc4204b6000)
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:390 +0x56
created by github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).Serve
    /home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:269 +0x4b3

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x45c2d3
rdx    0x0
rdi    0xf23f58
rsi    0x0
rbp    0x7f111938add8
rsp    0x7f111938ad90
r8     0x0
r9     0x0
r10    0x7f111938adc8
r11    0x246
r12    0x0
r13    0x7ffee299717f
r14    0x7f111938b9c0
r15    0x7ffee2997200
rip    0x45c2d3
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
*** Test killed: ran too long (10m0s).
FAIL    github.com/uber/zanzibar/test   2973.475s
ok      github.com/uber/zanzibar/test/clients/bar   0.945s
ok      github.com/uber/zanzibar/test/clients/baz   0.839s
ok      github.com/uber/zanzibar/test/config    0.005s
ok      github.com/uber/zanzibar/test/endpoints/bar 1.215s
ok      github.com/uber/zanzibar/test/endpoints/baz 1.354s
ok      github.com/uber/zanzibar/test/endpoints/contacts    0.054s
ok      github.com/uber/zanzibar/test/endpoints/googlenow   0.390s
ok      github.com/uber/zanzibar/test/endpoints/tchannel/baz    0.044s
?       github.com/uber/zanzibar/test/lib   [no test files]
?       github.com/uber/zanzibar/test/lib/bench_gateway [no test files]
?       github.com/uber/zanzibar/test/lib/test_backend  [no test files]
?       github.com/uber/zanzibar/test/lib/test_gateway  [no test files]
?       github.com/uber/zanzibar/test/lib/test_m3_server    [no test files]
?       github.com/uber/zanzibar/test/lib/util  [no test files]
Raynos commented 7 years ago

either /health_test.go:223 or the test block before that hangs. so TestRuntimeMetrics or TestHealthMetrics

Raynos commented 7 years ago 
raynos at raynos-ThinkPad-T440p  
~/gocode/src/github.com/uber/zanzibar on url-params-server-side
$ while ZANZIBAR_CACHE=1 go test ./test/ -run TestRuntimeMetrics ; do sleep 1 ; done
ok      github.com/uber/zanzibar/test   0.035s
ok      github.com/uber/zanzibar/test   0.047s
ok      github.com/uber/zanzibar/test   0.057s
ok      github.com/uber/zanzibar/test   0.036s

Confirmed it is TestRuntimeMetrics that hangs.