Closed tbg closed 8 years ago
The following test appears to have failed:
#20681:
I160728 19:19:35.941887 storage/replica.go:1660 store=2:2 ([]=) range=1 [/Min-/System/StatusNode/2): pending commands: refurbished 1, reproposing 0 (at 73.60); reasonTicks W160728 19:19:36.041048 kv/dist_sender.go:948 RPC error: rpc error: code = 4 desc = context deadline exceeded W160728 19:19:36.041130 kv/dist_sender.go:729 failed to invoke Put [/System/StatusNode/3,/Min): failed to send RPC: sending to all 2 replicas failed; last error: rpc error: code = 4 desc = context deadline exceeded W160728 19:19:36.041350 kv/dist_sender.go:948 RPC error: rpc error: code = 4 desc = context deadline exceeded W160728 19:19:36.041663 kv/dist_sender.go:729 failed to invoke Merge [/System/tsd/cr.node.gossip.infos.received-1m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.received-1h/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.sent-1m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.sent-10m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.sent-1h/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.connections.outgoing/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.received-1m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.sent-1h/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.sent-10m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.sent-count/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.received-10m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.sent-count/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.received-1h/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.received-count/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.infos.received-count/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.received-10m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.bytes.sent-1m/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.gossip.connections.incoming/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.sys.gc.count/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.node.sys.gc.pause.percent/3/10s/2016-07-28T19:00:00Z,/Min), ... 170 skipped ..., Merge [/System/tsd/cr.store.replicas/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.store.intentbytes/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.store.valcount/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.store.rocksdb.bloom.filter.prefix.useful/3/10s/2016-07-28T19:00:00Z,/Min), Merge [/System/tsd/cr.store.range.snapshots.preemptive-applied/3/10s/2016-07-28T19:00:00Z,/Min): failed to send RPC: sending to all 2 replicas failed; last error: rpc error: code = 4 desc = context deadline exceeded panic: test timed out after 2m0s goroutine 38654 [running]: panic(0x1689660, 0xc820805f80) /usr/local/go/src/runtime/panic.go:481 +0x3e6 testing.startAlarm.func1() /usr/local/go/src/testing/testing.go:725 +0x14b created by time.goFunc /usr/local/go/src/time/sleep.go:129 +0x3a goroutine 1 [chan receive]: testing.RunTests(0x1f259d0, 0x2747120, 0x60, 0x60, 0x5cc201) /usr/local/go/src/testing/testing.go:583 +0x8d2 testing.(*M).Run(0xc820033f08, 0x189ce20) /usr/local/go/src/testing/testing.go:515 +0x81 github.com/cockroachdb/cockroach/sql_test.TestMain(0xc820033f08) /go/src/github.com/cockroachdb/cockroach/sql/main_test.go:199 +0xe6 main.main() github.com/cockroachdb/cockroach/sql/_test/_testmain.go:604 +0x114 goroutine 17 [syscall, 1 minutes, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 goroutine 5 [chan receive]: github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x2a63400) /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1015 +0x64 created by github.com/cockroachdb/cockroach/util/log.init.1 /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:598 +0x8a goroutine 31771 [select]: github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc820ee28a0) /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4 github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm() /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820d2fdc0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31673 [select]: github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1() /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8211cf1e0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31670 [semacquire]: sync.runtime_Semacquire(0xc820912afc) /usr/local/go/src/runtime/sema.go:47 +0x26 sync.(*WaitGroup).Wait(0xc820912af0) /usr/local/go/src/sync/waitgroup.go:127 +0xb4 github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1() /go/src/github.com/cockroachdb/cockroach/storage/store.go:2582 +0x50d github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8211cf100) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31650 [chan receive]: github.com/cockroachdb/cockroach/server.(*Server).Start.func3() /go/src/github.com/cockroachdb/cockroach/server/server.go:340 +0x5e github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8202ba500) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31809 [select]: github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8204e3ce0) /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4 github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm() /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820caf4f0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31897 [chan receive]: github.com/cockroachdb/cmux.(*muxListener).Accept(0xc8203137e0, 0x0, 0x0, 0x0, 0x0) <autogenerated>:9 +0x125 net/http.(*Server).Serve(0xc820dd3d00, 0x7f3fde4a2580, 0xc8203137e0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2117 +0x129 github.com/cockroachdb/cockroach/server.(*Server).Start.func5() /go/src/github.com/cockroachdb/cockroach/server/server.go:356 +0x47 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820313860) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31807 [chan receive]: github.com/cockroachdb/cockroach/rpc.NewContext.func1() /go/src/github.com/cockroachdb/cockroach/rpc/context.go:105 +0x57 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8203125c0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31890 [select]: github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1() /go/src/github.com/cockroachdb/cockroach/sql/lease.go:1144 +0x1343 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8203e6090) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31780 [chan receive]: github.com/cockroachdb/cmux.(*muxListener).Accept(0xc820ed6840, 0x0, 0x0, 0x0, 0x0) <autogenerated>:9 +0x125 crypto/tls.(*listener).Accept(0xc820ed68a0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/tls.go:52 +0x60 net/http.(*Server).Serve(0xc82031b300, 0x7f3fde4a25b8, 0xc820ed68a0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2117 +0x129 github.com/cockroachdb/cockroach/server.(*Server).Start.func6() /go/src/github.com/cockroachdb/cockroach/server/server.go:363 +0x3a github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820ed68c0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31783 [chan receive]: github.com/cockroachdb/cockroach/gossip.(*server).start.func3() /go/src/github.com/cockroachdb/cockroach/gossip/server.go:317 +0x5c github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820ed69c0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31789 [IO wait]: net.runtime_pollWait(0x7f3fde4a2488, 0x72, 0xc821ffb000) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc82058b020, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc82058b020, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc82058afc0, 0xc821ffb000, 0x1000, 0x1000, 0x0, 0x7f3fdf64a050, 0xc82000e058) /usr/local/go/src/net/fd_unix.go:250 +0x23a net.(*conn).Read(0xc8202906b0, 0xc821ffb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:172 +0xe4 -- google.golang.org/grpc/transport.(*http2Client).reader(0xc82082c2d0) /go/src/google.golang.org/grpc/transport/http2_client.go:795 +0x109 created by google.golang.org/grpc/transport.newHTTP2Client /go/src/google.golang.org/grpc/transport/http2_client.go:174 +0xd21 goroutine 31681 [select]: github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1() /go/src/github.com/cockroachdb/cockroach/server/server.go:579 +0x15d github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8211cf420) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31756 [select]: google.golang.org/grpc/transport.(*http2Server).controller(0xc82062aab0) /go/src/google.golang.org/grpc/transport/http2_server.go:660 +0x5da created by google.golang.org/grpc/transport.newHTTP2Server /go/src/google.golang.org/grpc/transport/http2_server.go:134 +0x84f goroutine 31698 [semacquire]: sync.runtime_Semacquire(0xc8208cfd9c) /usr/local/go/src/runtime/sema.go:47 +0x26 sync.(*Mutex).Lock(0xc8208cfd98) /usr/local/go/src/sync/mutex.go:83 +0x1c4 github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc8208cfb80, 0x7f3fde4a37a8, 0xc82249bb90, 0x14658adc802ef7dd, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/storage/store.go:2189 +0x9a3 github.com/cockroachdb/cockroach/storage.(*Stores).Send(0xc8206484c0, 0x7f3fde4a37a8, 0xc82249bb90, 0x0, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/storage/stores.go:175 +0x3a0 github.com/cockroachdb/cockroach/server.(*Node).Batch.func3() /go/src/github.com/cockroachdb/cockroach/server/node.go:794 +0x558 -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820d207e0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31684 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820c8aba0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31695 [IO wait]: net.runtime_pollWait(0x7f3fde4a1e88, 0x72, 0xc820dce800) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc8208796b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc8208796b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc820879650, 0xc820dce800, 0x400, 0x400, 0x0, 0x7f3fdf64a050, 0xc82000e058) /usr/local/go/src/net/fd_unix.go:250 +0x23a net.(*conn).Read(0xc820290370, 0xc820dce800, 0x400, 0x400, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:172 +0xe4 -- google.golang.org/grpc/transport.(*http2Client).reader(0xc82012cd20) /go/src/google.golang.org/grpc/transport/http2_client.go:795 +0x109 created by google.golang.org/grpc/transport.newHTTP2Client /go/src/google.golang.org/grpc/transport/http2_client.go:174 +0xd21 goroutine 31692 [select]: github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc8208789a0, 0xecf2c5193, 0x6e2566, 0x2a622a0, 0xc82036ac80, 0xc8203bb680, 0xc820576e00, 0xc820b47698) /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc820576e00, 0xc820afdc50) /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:214 +0x78 github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc820afdf20, 0xc820a636c0) /go/src/github.com/cockroachdb/cockroach/storage/store.go:283 +0x246 github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1() /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:216 +0x21e github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820c8b180) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31832 [select]: google.golang.org/grpc/transport.(*http2Server).controller(0xc820566750) /go/src/google.golang.org/grpc/transport/http2_server.go:660 +0x5da created by google.golang.org/grpc/transport.newHTTP2Server /go/src/google.golang.org/grpc/transport/http2_server.go:134 +0x84f goroutine 31945 [select]: github.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1() /go/src/github.com/cockroachdb/cockroach/server/node.go:605 +0x2df github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820c8a980) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31896 [IO wait]: net.runtime_pollWait(0x7f3fde4a1c48, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc820a111e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820a111e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).accept(0xc820a11180, 0x0, 0x7f3fdf669150, 0xc820313ce0) /usr/local/go/src/net/fd_unix.go:426 +0x27c net.(*TCPListener).AcceptTCP(0xc8202902a0, 0x60ec00, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820313820) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31685 [semacquire]: sync.runtime_Semacquire(0xc820576e74) /usr/local/go/src/runtime/sema.go:47 +0x26 sync.(*Mutex).Lock(0xc820576e70) /usr/local/go/src/sync/mutex.go:83 +0x1c4 github.com/cockroachdb/cockroach/storage.(*Replica).proposeRaftCommand.func1(0xc821d5ad58) /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1383 +0x42 github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc820576e00, 0x7f3fde4a37a8, 0xc8207d5230, 0x14658ada57decca2, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc8206060c0, ...) /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1296 +0x567 github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc820576e00, 0x7f3fde4a37a8, 0xc8207d5230, 0x14658ada57decca2, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc8206060c0, ...) /go/src/github.com/cockroachdb/cockroach/storage/replica.go:827 +0x23d -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820c8abe0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31759 [select]: google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc8209898f0, 0xc8206bd310, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/google.golang.org/grpc/transport/transport.go:141 +0x7e6 google.golang.org/grpc/transport.(*Stream).Read(0xc8203cf6c0, 0xc8206bd310, 0x5, 0x5, 0x844548, 0x0, 0x0) /go/src/google.golang.org/grpc/transport/transport.go:294 +0x71 io.ReadAtLeast(0x7f3fde4a6020, 0xc8203cf6c0, 0xc8206bd310, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:297 +0xe6 io.ReadFull(0x7f3fde4a6020, 0xc8203cf6c0, 0xc8206bd310, 0x5, 0x5, 0xc800000002, 0x0, 0x0) /usr/local/go/src/io/io.go:315 +0x62 google.golang.org/grpc.(*parser).recvMsg(0xc8206bd300, 0x9b279bab30f12ee6, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/google.golang.org/grpc/rpc_util.go:216 +0xb9 -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820989980) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31645 [chan receive]: github.com/cockroachdb/cockroach/rpc.NewContext.func1() /go/src/github.com/cockroachdb/cockroach/rpc/context.go:105 +0x57 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820b759e0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31820 [select]: github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1() /go/src/github.com/cockroachdb/cockroach/server/server.go:579 +0x15d github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820610460) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31701 [IO wait]: net.runtime_pollWait(0x7f3fde4a2248, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc820692060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820692060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).accept(0xc820692000, 0x0, 0x7f3fdf669150, 0xc820e5e3e0) /usr/local/go/src/net/fd_unix.go:426 +0x27c net.(*TCPListener).AcceptTCP(0xc8202900c8, 0xc820b6eed8, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8211cf4c0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31822 [chan receive]: github.com/cockroachdb/cockroach/kv.(*rangeDescriptorCache).lookupRangeDescriptorInternal(0xc820ebe280, 0x7f3fde4a37a8, 0xc82256cae0, 0xc8227281a0, 0xe, 0x16, 0xc822bd7b40, 0x16b0001, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/kv/range_cache.go:337 +0x9df github.com/cockroachdb/cockroach/kv.(*rangeDescriptorCache).LookupRangeDescriptor(0xc820ebe280, 0x7f3fde4a37a8, 0xc82256cae0, 0xc8227281a0, 0xe, 0x16, 0xc822bd7b40, 0x1, 0x1bd53a0, 0x16, ...) /go/src/github.com/cockroachdb/cockroach/kv/range_cache.go:283 +0xa3 github.com/cockroachdb/cockroach/kv.(*DistSender).getDescriptors(0xc82035eea0, 0x7f3fde4a37a8, 0xc82256cae0, 0xc8227281a0, 0xe, 0x16, 0xc8227281a0, 0xf, 0x16, 0xc822bd7b40, ...) /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:410 +0xd2 github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk(0xc82035eea0, 0x7f3fde4a37a8, 0xc82256cae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:646 +0x3e0 github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc82035eea0, 0x7f3fde4a37a8, 0xc82256cae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:565 +0x6d4 -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820610480) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31787 [select]: github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc82085bbc0, 0xc820c44790, 0xc8208b0320, 0xf, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x68d github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1() /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8202d7950) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31778 [IO wait]: net.runtime_pollWait(0x7f3fde4a1f48, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc82058ad80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc82058ad80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).accept(0xc82058ad20, 0x0, 0x7f3fdf669150, 0xc820e5e7c0) /usr/local/go/src/net/fd_unix.go:426 +0x27c net.(*TCPListener).AcceptTCP(0xc820290640, 0x60ec00, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d -- github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820ed6860) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31772 [select]: github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1() /go/src/github.com/cockroachdb/cockroach/sql/lease.go:1144 +0x1343 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8202d6900) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31668 [select]: github.com/cockroachdb/cockroach/storage.(*bookie).start.func1() /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:257 +0x262 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8211ceac0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31714 [select]: github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc82045bc80, 0xc820465ef0, 0xc820730330, 0xf, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x68d github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1() /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8207c4060) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31833 [IO wait]: net.runtime_pollWait(0x7f3fde4a20c8, 0x72, 0xc820514000) /usr/local/go/src/runtime/netpoll.go:160 +0x60 net.(*pollDesc).Wait(0xc820575d40, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820575d40, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc820575ce0, 0xc820514000, 0x400, 0x400, 0x0, 0x7f3fdf64a050, 0xc82000e058) /usr/local/go/src/net/fd_unix.go:250 +0x23a net.(*conn).Read(0xc820290918, 0xc820514000, 0x400, 0x400, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:172 +0xe4 -- google.golang.org/grpc/transport.(*http2Client).reader(0xc820b35a40) /go/src/google.golang.org/grpc/transport/http2_client.go:795 +0x109 created by google.golang.org/grpc/transport.newHTTP2Client /go/src/google.golang.org/grpc/transport/http2_client.go:174 +0xd21 goroutine 31786 [select]: google.golang.org/grpc.(*Conn).transportMonitor(0xc820da9c20) /go/src/google.golang.org/grpc/clientconn.go:547 +0x1d3 google.golang.org/grpc.NewConn.func1(0xc820da9c20) /go/src/google.golang.org/grpc/clientconn.go:355 +0x1b5 created by google.golang.org/grpc.NewConn /go/src/google.golang.org/grpc/clientconn.go:356 +0x4e3 goroutine 32557 [select]: google.golang.org/grpc.(*Conn).transportMonitor(0xc822019590) /go/src/google.golang.org/grpc/clientconn.go:547 +0x1d3 created by google.golang.org/grpc.NewConn /go/src/google.golang.org/grpc/clientconn.go:346 +0x49f goroutine 31655 [chan receive]: github.com/cockroachdb/cmux.(*muxListener).Accept(0xc8202ba400, 0x0, 0x0, 0x0, 0x0) <autogenerated>:9 +0x125 github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc8202900c0, 0xc8203bb680, 0x7f3fde4a2580, 0xc8202ba400, 0xc820730c90, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:112 +0x67 github.com/cockroachdb/cockroach/server.(*Server).Start.func8() /go/src/github.com/cockroachdb/cockroach/server/server.go:375 +0xac github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8206f0f60) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31774 [chan receive]: github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2() /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:95 +0x60 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc820ed65a0) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31632 [chan receive]: github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2() /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:95 +0x60 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8202ba300) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31782 [chan receive]: github.com/cockroachdb/cmux.(*muxListener).Accept(0xc820ed6740, 0x0, 0x0, 0x0, 0x0) <autogenerated>:9 +0x125 github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc8202905f0, 0xc8203bb680, 0x7f3fde4a2580, 0xc820ed6740, 0xc820d21870, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:112 +0x67 github.com/cockroachdb/cockroach/server.(*Server).Start.func8() /go/src/github.com/cockroachdb/cockroach/server/server.go:375 +0xac github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8203bb680, 0xc8202d7530) /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62 goroutine 31753 [select]: github.com/cockroachdb/cockroach/storage.(*bookie).start.func1() /go/src/github.com/cockroachdb/cockroach/storage/rese
Please assign, take a look and update the issue accordingly.
Duplicate of #8098.
petermattis
commented
8 years ago petermattis
commented
8 years ago
The following test appears to have failed:
#20681:
Please assign, take a look and update the issue accordingly.