Closed tbg closed 8 years ago
make stress RANDOM_SEED=3682831729549554501 PKG=./storage TESTS='TestReplicateRogueRemovedNode'
Pretty sure the random seed doesn't matter, but that's how I happened to call it. 2 failures in ~7.5k runs.
I1209 11:57:32.877835 69635 rand.go:87 Random seed: 3682831729549554501 E1209 11:57:32.879375 69635 gossip/gossip.go:181 different node IDs were set for the same gossip instance (2147483647, 1) I1209 11:57:32.880175 69635 multiraft/multiraft.go:999 node 1 campaigning because initial confstate is [1] I1209 11:57:32.880698 69635 storage/replica_command.go:1208 range 1: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC +1.000s I1209 11:57:32.881059 69635 gossip/gossip.go:189 gossiping node descriptor node_id:1 address:<network_field:"localhost" address_field:"1" > attrs:<> E1209 11:57:32.881413 69635 gossip/gossip.go:181 different node IDs were set for the same gossip instance (1, 2) I1209 11:57:32.881620 69635 gossip/gossip.go:189 gossiping node descriptor node_id:2 address:<network_field:"localhost" address_field:"2" > attrs:<> E1209 11:57:32.881962 69635 gossip/gossip.go:181 different node IDs were set for the same gossip instance (2, 3) I1209 11:57:32.882119 69635 gossip/gossip.go:189 gossiping node descriptor node_id:3 address:<network_field:"localhost" address_field:"3" > attrs:<> I1209 11:57:32.883130 69635 storage/store.go:1544 raft: ADD_REPLICA {2 2 2} for range 1 I1209 11:57:32.884297 69635 storage/store.go:1544 raft: ADD_REPLICA {3 3 3} for range 1 E1209 11:57:33.176231 69635 storage/client_test.go:558 engine 1: missing key "a" E1209 11:57:33.176281 69635 storage/client_test.go:558 engine 2: missing key "a" I1209 11:57:33.179095 69635 storage/store.go:1544 raft: REMOVE_REPLICA {3 3 3} for range 1 W1209 11:57:33.179950 69635 multiraft/multiraft.go:1298 node 1 failed to send message to 3: multiraft/transport.go:143: unknown peer 3 W1209 11:57:33.180237 69635 multiraft/multiraft.go:1298 node 1 failed to send message to 3: multiraft/transport.go:143: unknown peer 3 W1209 11:57:33.181384 69635 multiraft/multiraft.go:1298 node 1 failed to send message to 3: multiraft/transport.go:143: unknown peer 3 I1209 11:57:33.182500 69635 storage/store.go:1544 raft: REMOVE_REPLICA {2 2 2} for range 1 W1209 11:57:33.184899 69635 multiraft/multiraft.go:1298 node 2 failed to send message to 3: multiraft/transport.go:143: unknown peer 3 E1209 11:57:33.185309 69635 raft/log.go:311 [group 1] entries[17:18) is unavailable from storage panic: [group 1] entries[17:18) is unavailable from storage goroutine 43 [running]: github.com/cockroachdb/cockroach/multiraft.(*raftLogger).Panicf(0xc82036e990, 0x504a960, 0x2a, 0xc820487d40, 0x2, 0x2) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/raft.go:121 +0x1a2 github.com/coreos/etcd/raft.(*raftLog).slice(0xc820014700, 0x11, 0x12, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/coreos/etcd/raft/log.go:311 +0x460 github.com/coreos/etcd/raft.(*raftLog).nextEnts(0xc820014700, 0x0, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/coreos/etcd/raft/log.go:142 +0x9f github.com/coreos/etcd/raft.newReady(0xc820166540, 0xc82036ec10, 0x6, 0x0, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /Users/tschottdorf/go/src/github.com/coreos/etcd/raft/node.go:450 +0x6f github.com/coreos/etcd/raft.(*RawNode).newReady(0xc82026dd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /Users/tschottdorf/go/src/github.com/coreos/etcd/raft/rawnode.go:40 +0x5f github.com/coreos/etcd/raft.(*RawNode).Ready(0xc82026dd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /Users/tschottdorf/go/src/github.com/coreos/etcd/raft/rawnode.go:182 +0x42 github.com/cockroachdb/cockroach/multiraft.(*state).handleWriteReady(0xc820133630, 0xc82046f140, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:1129 +0x501 github.com/cockroachdb/cockroach/multiraft.(*state).start.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:654 +0x1766 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8201589c0, 0xc820260290) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 1 [chan receive]: testing.RunTests(0x512f0a8, 0x54a0420, 0xda, 0xda, 0x1) /usr/local/Cellar/go/1.5.1/libexec/src/testing/testing.go:562 +0x8ad testing.(*M).Run(0xc8200d3ee8, 0x1) /usr/local/Cellar/go/1.5.1/libexec/src/testing/testing.go:494 +0x70 github.com/cockroachdb/cockroach/util/leaktest.TestMainWithLeakCheck(0xc8200d3ee8) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:34 +0x21 github.com/cockroachdb/cockroach/storage_test.TestMain(0xc8200d3ee8) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/main_test.go:37 +0x26 main.main() github.com/cockroachdb/cockroach/storage/_test/_testmain.go:496 +0x113 goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1 goroutine 20 [chan receive]: github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x5505a00) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1029 +0x64 created by github.com/cockroachdb/cockroach/util/log.init.1 /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:610 +0xbc goroutine 22 [runnable, locked to thread]: github.com/cockroachdb/cockroach/storage/engine._Cfunc_DBIterDestroy(0x8305000) ??:0 +0x31 github.com/cockroachdb/cockroach/storage/engine.(*rocksDBIterator).Close(0xc820278b70) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:518 +0x24 github.com/cockroachdb/cockroach/storage/engine.MVCCGet(0x8267c20, 0xc8201a5360, 0xc820461cd0, 0x1, 0x8, 0x311cde8b0ca01, 0xc, 0x1, 0x0, 0xc8204b0450, ...) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:508 +0x4e9 github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).readIntFromEngines(0xc8200b97c0, 0xc820461cd0, 0x1, 0x8, 0x0, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:554 +0x1be github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).waitForValues.func1(0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:574 +0x69 github.com/cockroachdb/cockroach/util.SucceedsWithinDepth(0x1, 0x6285178, 0xc8200a6e10, 0x3b9aca00, 0xc8202cbc70) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/testing.go:148 +0x18f github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).waitForValues(0xc8200b97c0, 0xc820461cd0, 0x1, 0x8, 0x3b9aca00, 0xc820486cc0, 0x3, 0x3) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:579 +0xcd github.com/cockroachdb/cockroach/storage_test.TestReplicateRogueRemovedNode(0xc8200a6e10) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/client_raft_test.go:1322 +0xaf1 testing.tRunner(0xc8200a6e10, 0x54a1650) /usr/local/Cellar/go/1.5.1/libexec/src/testing/testing.go:456 +0x98 created by testing.RunTests /usr/local/Cellar/go/1.5.1/libexec/src/testing/testing.go:561 +0x86d goroutine 35 [select]: github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:203 +0xd2f github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200759e0, 0xc82013a7a0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 23 [select]: github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:214 +0x1b6 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc820186a40) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 24 [select]: github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8201a2500) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:215 +0xca5 github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:202 +0x20 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc8201a1a40) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 25 [chan receive]: github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc8201a5360) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:125 +0x3a created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:126 +0x4a8 goroutine 26 [IO wait]: net.runtime_pollWait(0x826d5a8, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc820181790, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820181790, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).accept(0xc820181730, 0x0, 0x82a0268, 0xc8202d2540) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:408 +0x27c net.(*TCPListener).AcceptTCP(0xc820096360, 0xc8201f8e30, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/tcpsock_posix.go:254 +0x4d net.(*TCPListener).Accept(0xc820096360, 0x0, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/tcpsock_posix.go:264 +0x3d net/http.(*Server).Serve(0xc8201f22a0, 0x826c600, 0xc820096360, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1887 +0xb3 github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:259 +0x3b github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc820186f40) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 27 [chan receive]: github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:265 +0x4b github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc8201e62b0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 28 [select]: github.com/cockroachdb/cockroach/multiraft.(*state).start.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:608 +0x2586 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200759e0, 0xc8201e6620) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 29 [select]: github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:1562 +0x11ac github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200759e0, 0xc8201e6630) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 30 [select]: github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:583 +0x3bf github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200759e0, 0xc820187040) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 31 [select]: github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:602 +0x3bf github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200759e0, 0xc820187060) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 50 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc8201870c0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 51 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc8201870e0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 52 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc820187100) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 53 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc820187120) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 54 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc820187140) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 55 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc820187160) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 56 [select]: github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820075b60, 0xecdfa547c, 0x3471d45d, 0x55055a0, 0xc820193b60, 0xc820075560, 0xc8200ca1c0, 0xc800000001) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:177 +0x53f github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc8200ca1c0, 0xc8201e4150) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:218 +0x67 github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc8201e41b0, 0xc8201e5200) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:211 +0x246 github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:220 +0x29a github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc820187180) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 12 [IO wait]: net.runtime_pollWait(0x826d1e8, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc820306ca0, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820306ca0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc820306c40, 0xc820366000, 0x1000, 0x1000, 0x0, 0x8260028, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a net.(*conn).Read(0xc8202d8040, 0xc820366000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4 bufio.(*Reader).fill(0xc820018ea0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9 bufio.(*Reader).ReadByte(0xc820018ea0, 0x1, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:229 +0x7a encoding/binary.ReadUvarint(0x87404a8, 0xc820018ea0, 0xa, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/encoding/binary/varint.go:110 +0x52 github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820001c80, 0x82a00c8, 0xc820001db8, 0xc800000000, 0x512d5d8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x6d github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820001c80, 0xc820001db8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x7d github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820001c80, 0xc8202b0f30, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x4b net/rpc.(*Client).input(0xc820018f00) /usr/local/Cellar/go/1.5.1/libexec/src/net/rpc/client.go:109 +0xbf created by net/rpc.NewClientWithCodec /usr/local/Cellar/go/1.5.1/libexec/src/net/rpc/client.go:201 +0xd2 goroutine 99 [IO wait]: net.runtime_pollWait(0x826d128, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc8202cc290, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc8202cc290, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc8202cc230, 0xc82035b000, 0x1000, 0x1000, 0x0, 0x8260028, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a net.(*conn).Read(0xc8202d8048, 0xc82035b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4 bufio.(*Reader).fill(0xc82029a4e0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9 bufio.(*Reader).ReadByte(0xc82029a4e0, 0x434a54d, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:229 +0x7a encoding/binary.ReadUvarint(0x87404a8, 0xc82029a4e0, 0xc8204a0b40, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/encoding/binary/varint.go:110 +0x52 github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820286300, 0x82a0090, 0xc820286458, 0xc800000000, 0x512d5d8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x6d github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc820286300, 0xc82029a4e0, 0xc820286458, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x7d github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc820286300, 0xc820487480, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x65 github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc8200b5b80, 0x8740468, 0xc820286300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:404 +0xbf github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc8200b5b80, 0x87402a0, 0xc8202d8048, 0x8740468, 0xc820286300, 0xc8202bc9a0, 0xc82029a540) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:360 +0x105 github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc8200b5b80, 0x87403a8, 0xc8202d6580, 0xc8202da380) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:203 +0x895 net/http.serverHandler.ServeHTTP(0xc8201f22a0, 0x87403a8, 0xc8202d6580, 0xc8202da380) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1862 +0x19e net/http.(*conn).serve(0xc8202d64d0) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1361 +0xbee created by net/http.(*Server).Serve /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1910 +0x3f6 goroutine 10 [runnable]: net.runtime_pollWait(0x826d368, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc820306ae0, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc820306ae0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc820306a80, 0xc820348000, 0x1000, 0x1000, 0x0, 0x8260028, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a net.(*conn).Read(0xc8202fe040, 0xc820348000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4 bufio.(*Reader).fill(0xc820018d80) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9 bufio.(*Reader).ReadByte(0xc820018d80, 0x434a54d, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:229 +0x7a encoding/binary.ReadUvarint(0x87404a8, 0xc820018d80, 0xa, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/encoding/binary/varint.go:110 +0x52 github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820001b00, 0x82a00c8, 0xc820001c38, 0xc800000000, 0x512d5d8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x6d github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820001b00, 0xc820001c38, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x7d github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820001b00, 0xc8202b0d20, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x4b net/rpc.(*Client).input(0xc820018de0) /usr/local/Cellar/go/1.5.1/libexec/src/net/rpc/client.go:109 +0xbf created by net/rpc.NewClientWithCodec /usr/local/Cellar/go/1.5.1/libexec/src/net/rpc/client.go:201 +0xd2 goroutine 40 [chan receive]: github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc820133450) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:125 +0x3a created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:126 +0x4a8 goroutine 41 [runnable]: net.runtime_pollWait(0x826d4e8, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc82021adf0, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc82021adf0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).accept(0xc82021ad90, 0x0, 0x82a0268, 0xc8202d2460) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:408 +0x27c net.(*TCPListener).AcceptTCP(0xc820138238, 0xc82003ce30, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/tcpsock_posix.go:254 +0x4d net.(*TCPListener).Accept(0xc820138238, 0x0, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/tcpsock_posix.go:264 +0x3d net/http.(*Server).Serve(0xc8201591a0, 0x826c600, 0xc820138238, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1887 +0xb3 github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:259 +0x3b github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc82013b6c0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 42 [chan receive]: github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:265 +0x4b github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc820260240) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 44 [select]: github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:1562 +0x11ac github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8201589c0, 0xc8202602a0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 7 [select]: github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:203 +0xd2f github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8201589c0, 0xc82000f060) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 46 [select]: github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:583 +0x3bf github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8201589c0, 0xc82013b6e0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 47 [select]: github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/store.go:602 +0x3bf github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8201589c0, 0xc82013b700) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 98 [runnable]: net.runtime_pollWait(0x826d2a8, 0x72, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60 net.(*pollDesc).Wait(0xc8202cc220, 0x72, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a net.(*pollDesc).WaitRead(0xc8202cc220, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36 net.(*netFD).Read(0xc8202cc1c0, 0xc820341000, 0x1000, 0x1000, 0x0, 0x8260028, 0xc8200920b0) /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a net.(*conn).Read(0xc8202d8038, 0xc820341000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4 bufio.(*Reader).fill(0xc82030e720) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9 bufio.(*Reader).ReadByte(0xc82030e720, 0x2, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:229 +0x7a encoding/binary.ReadUvarint(0x87404a8, 0xc82030e720, 0xc8204a0d80, 0x0, 0x0) /usr/local/Cellar/go/1.5.1/libexec/src/encoding/binary/varint.go:110 +0x52 github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820332180, 0x82a0090, 0xc8203322d8, 0xc800000000, 0x512d5d8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x6d github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc820332180, 0xc82030e720, 0xc8203322d8, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x7d github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc820332180, 0xc8204874c0, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x65 github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc82013c700, 0x8740468, 0xc820332180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:404 +0xbf github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc82013c700, 0x87402a0, 0xc8202d8038, 0x8740468, 0xc820332180, 0xc8202f4f80, 0xc82030e780) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:360 +0x105 github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc82013c700, 0x87403a8, 0xc82031c370, 0xc8202da1c0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:203 +0x895 net/http.serverHandler.ServeHTTP(0xc8201591a0, 0x87403a8, 0xc82031c370, 0xc8202da1c0) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1862 +0x19e net/http.(*conn).serve(0xc8202d6420) /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1361 +0xbee created by net/http.(*Server).Serve /usr/local/Cellar/go/1.5.1/libexec/src/net/http/server.go:1910 +0x3f6 goroutine 49 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b740) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 66 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b760) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 67 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b780) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 68 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b7a0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 69 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b7c0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 70 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b7e0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 71 [select]: github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820158b40, 0xecdfa547c, 0x348b8024, 0x55055a0, 0xc820193b60, 0xc820075560, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:177 +0x53f github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:214 +0x138 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013b800) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 73 [chan receive]: github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc820133770) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:125 +0x3a created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:126 +0x4a8 goroutine 75 [chan receive]: github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:265 +0x4b github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075680, 0xc820260820) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 82 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bc20) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 83 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bc40) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 84 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bc60) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 85 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bc80) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 86 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bca0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 87 [select]: github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/queue.go:287 +0x329 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bcc0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 88 [select]: github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820159800, 0xecdfa547c, 0x34930c03, 0x55055a0, 0xc820193b60, 0xc820075560, 0x0, 0x0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:177 +0x53f github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1() /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:214 +0x138 github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820075560, 0xc82013bce0) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x52 created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x62 goroutine 114 [chan receive]: github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc82013c700, 0x8740468, 0xc820332180, 0xc82030e780) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:427 +0xa4 github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc82013c700, 0x8740468, 0xc820332180, 0xc82030e780) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:201 +0x3f created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:202 +0x83c goroutine 100 [chan receive]: github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc8200b5b80, 0x8740468, 0xc820286300, 0xc82029a540) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:427 +0xa4 github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc8200b5b80, 0x8740468, 0xc820286300, 0xc82029a540) /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:201 +0x3f created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/rpc/server.go:202 +0x83c ERROR: exit status 2
No longer reproducible
bdarnell
commented
8 years ago bdarnell
commented
8 years ago
make stress RANDOM_SEED=3682831729549554501 PKG=./storage TESTS='TestReplicateRogueRemovedNode'Pretty sure the random seed doesn't matter, but that's how I happened to call it. 2 failures in ~7.5k runs.