stress: failed test in cockroach/testutils/testcluster/testcluster.test: TestManualReplication

Binary: cockroach/static-tests.tar.gz sha: https://github.com/cockroachdb/cockroach/commits/4a0d487e07434360add95e2a33341c930d442a94

Stress build found a failed test:

=== RUN   TestManualReplication
I160807 05:16:55.900602 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160807 05:16:55.931660 gossip/gossip.go:1015  not connected to cluster; use --join to specify a connected node
I160807 05:16:55.932789 server/node.go:403  store store=0:0 not bootstrapped
I160807 05:16:55.936356 storage/replica_trigger.go:308  store=1:1 range=1 [/Min-/Max): new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 408485h17m0.435619278s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=2016-08-07 05:16:55.936245297 +0000 UTC]
I160807 05:16:55.949999 server/node.go:333  **** cluster {f6f4b9e0-06e9-4379-a2e7-425f3029ee5e} has been created
I160807 05:16:55.950045 server/node.go:334  **** add additional nodes by specifying --join=127.0.0.1:40290
I160807 05:16:55.950783 storage/store.go:2696  system config not yet available
I160807 05:16:55.950838 server/node.go:416  initialized store store=1:1: {Capacity:536870912 Available:536870912 RangeCount:0}
I160807 05:16:55.950859 server/node.go:306  node ID 1 initialized
I160807 05:16:55.950918 storage/stores.go:302  read 0 node addresses from persistent storage
I160807 05:16:55.951020 server/node.go:541  connecting to gossip network to verify cluster ID...
I160807 05:16:55.951380 server/node.go:562  node connected via gossip and verified as part of cluster {"f6f4b9e0-06e9-4379-a2e7-425f3029ee5e"}
I160807 05:16:55.951419 server/node.go:355  node=1: started with [[]=] engine(s) and attributes []
I160807 05:16:55.951452 server/server.go:432  starting https server at 127.0.0.1:41375
I160807 05:16:55.951466 server/server.go:433  starting grpc/postgres server at 127.0.0.1:40290
I160807 05:16:55.951706 sql/event_log.go:109  Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:40290} Attrs:} ClusterID:{UUID:f6f4b9e0-06e9-4379-a2e7-425f3029ee5e} StartedAt:1470547015951386908}
I160807 05:16:55.952466 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160807 05:16:55.954712 sql/event_log.go:109  Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:40290} Attrs:} ClusterID:{UUID:f6f4b9e0-06e9-4379-a2e7-425f3029ee5e} StartedAt:1470547015951386908}
W160807 05:16:55.978692 gossip/gossip.go:1015  not connected to cluster; use --join to specify a connected node
I160807 05:16:56.001065 gossip/client.go:75  starting client to 127.0.0.1:40290
I160807 05:16:56.001221 server/node.go:403  store store=0:0 not bootstrapped
I160807 05:16:56.001249 storage/stores.go:302  read 0 node addresses from persistent storage
I160807 05:16:56.001277 server/node.go:541  connecting to gossip network to verify cluster ID...
I160807 05:16:56.027187 server/node.go:562  node connected via gossip and verified as part of cluster {"f6f4b9e0-06e9-4379-a2e7-425f3029ee5e"}
I160807 05:16:56.030336 storage/stores.go:317  wrote 1 node addresses to persistent storage
I160807 05:16:56.032691 kv/dist_sender.go:329  unable to determine this node's attributes for replica selection; node is most likely bootstrapping
I160807 05:16:56.033534 server/node.go:300  new node allocated ID 2
I160807 05:16:56.033622 server/node.go:355  node=2: started with [[]=] engine(s) and attributes []
I160807 05:16:56.033675 server/server.go:432  starting https server at 127.0.0.1:33084
I160807 05:16:56.033693 server/server.go:433  starting grpc/postgres server at 127.0.0.1:34816
I160807 05:16:56.033771 sql/event_log.go:109  Event: "node_join", target: 2, info: {Descriptor:{NodeID:2 Address:{NetworkField:tcp AddressField:127.0.0.1:34816} Attrs:} ClusterID:{UUID:f6f4b9e0-06e9-4379-a2e7-425f3029ee5e} StartedAt:1470547016033601254}
I160807 05:16:56.035061 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160807 05:16:56.035425 storage/stores.go:317  wrote 1 node addresses to persistent storage
I160807 05:16:56.042887 server/node.go:522  bootstrapped store store=2:2
W160807 05:16:56.044097 gossip/gossip.go:1015  not connected to cluster; use --join to specify a connected node
I160807 05:16:56.071440 server/node.go:403  store store=0:0 not bootstrapped
I160807 05:16:56.071465 storage/stores.go:302  read 0 node addresses from persistent storage
I160807 05:16:56.071486 server/node.go:541  connecting to gossip network to verify cluster ID...
I160807 05:16:56.071607 gossip/client.go:75  starting client to 127.0.0.1:40290
I160807 05:16:56.088490 server/node.go:562  node connected via gossip and verified as part of cluster {"f6f4b9e0-06e9-4379-a2e7-425f3029ee5e"}
I160807 05:16:56.088808 storage/stores.go:317  wrote 1 node addresses to persistent storage
I160807 05:16:56.088840 storage/stores.go:317  wrote 2 node addresses to persistent storage
I160807 05:16:56.090672 kv/dist_sender.go:329  unable to determine this node's attributes for replica selection; node is most likely bootstrapping
I160807 05:16:56.091339 server/node.go:300  new node allocated ID 3
I160807 05:16:56.091398 server/node.go:355  node=3: started with [[]=] engine(s) and attributes []
I160807 05:16:56.094503 server/server.go:432  starting https server at 127.0.0.1:43238
I160807 05:16:56.094525 server/server.go:433  starting grpc/postgres server at 127.0.0.1:45158
I160807 05:16:56.094694 server/node.go:522  bootstrapped store store=3:3
I160807 05:16:56.094849 sql/event_log.go:109  Event: "node_join", target: 3, info: {Descriptor:{NodeID:3 Address:{NetworkField:tcp AddressField:127.0.0.1:45158} Attrs:} ClusterID:{UUID:f6f4b9e0-06e9-4379-a2e7-425f3029ee5e} StartedAt:1470547016091381430}
I160807 05:16:56.095159 storage/stores.go:317  wrote 2 node addresses to persistent storage
I160807 05:16:56.095645 storage/stores.go:317  wrote 2 node addresses to persistent storage
panic: close of closed channel [recovered]
    panic: close of closed channel

goroutine 250 [running]:
panic(0x149d4a0, 0xc420974870)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).Recover(0xc42027e090)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:176 +0x6e
panic(0x149d4a0, 0xc420974870)
    /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/cockroachdb/cockroach/testutils/testcluster.(*TestCluster).waitForStores.func1(0xc4209387f3, 0x7, 0xc4203ad300, 0x37, 0x40, 0x14686eb847d62a51, 0x0)
    /go/src/github.com/cockroachdb/cockroach/testutils/testcluster/testcluster.go:180 +0x16c
github.com/cockroachdb/cockroach/gossip.(*infoStore).runCallbacks.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:266 +0xa5
github.com/cockroachdb/cockroach/gossip.(*infoStore).runCallbacks.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:288 +0xdf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc42027e090, 0x1badc9b, 0x13, 0x122, 0xc4201b1450)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201e40c0, 0x15d4f93, 0x15, 0x16ee400, 0xc4202e1ce0)
    /usr/local/go/src/testing/testing.go:647 +0x316
testing.RunTests.func1(0xc4201e40c0)
    /usr/local/go/src/testing/testing.go:793 +0x6d
testing.tRunner(0xc4201e40c0, 0xc420344e00)
    /usr/local/go/src/testing/testing.go:610 +0x81
testing.RunTests(0x16ef370, 0x1eb2b80, 0x4, 0x4, 0x223f610)
    /usr/local/go/src/testing/testing.go:799 +0x2f5
testing.(*M).Run(0xc420344ef8, 0x223f200)
    /usr/local/go/src/testing/testing.go:743 +0x85
github.com/cockroachdb/cockroach/testutils/testcluster.TestMain(0xc420344ef8)
    /go/src/github.com/cockroachdb/cockroach/testutils/testcluster/main_test.go:34 +0x92
main.main()
    github.com/cockroachdb/cockroach/testutils/testcluster/_test/_testmain.go:58 +0xc6

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x21f1dc0)
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1015 +0x77
created by github.com/cockroachdb/cockroach/util/log.init.1
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:598 +0x95

goroutine 51 [runnable]:
syscall.Syscall(0x2a, 0x14, 0xc420a4a00c, 0x10, 0xffffffffffffffff, 0x0, 0x73)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.connect(0x14, 0xc420a4a00c, 0xc400000010, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1614 +0x50
syscall.Connect(0x14, 0x1ec30c0, 0xc420a4a000, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:229 +0x64
net.(*netFD).connect(0xc420a48000, 0x1ed04c0, 0xc4200105a0, 0x0, 0x0, 0x1ec30c0, 0xc420a4a000, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:71 +0x89
net.(*netFD).dial(0xc420a48000, 0x1ed04c0, 0xc4200105a0, 0x1ed2fa0, 0x0, 0x1ed2fa0, 0xc4201aa7b0, 0x10, 0x10)
    /usr/local/go/src/net/sock_posix.go:137 +0x137
net.socket(0x1ed04c0, 0xc4200105a0, 0x15c3bc4, 0x3, 0x2, 0x1, 0x0, 0x0, 0x1ed2fa0, 0x0, ...)
    /usr/local/go/src/net/sock_posix.go:89 +0x209
net.internetSocket(0x1ed04c0, 0xc4200105a0, 0x15c3bc4, 0x3, 0x1ed2fa0, 0x0, 0x1ed2fa0, 0xc4201aa7b0, 0x1, 0x0, ...)
    /usr/local/go/src/net/ipsock_posix.go:158 +0x129
net.doDialTCP(0x1ed04c0, 0xc4200105a0, 0x15c3bc4, 0x3, 0x0, 0xc4201aa7b0, 0x149d1a0, 0x223f610, 0xc4204eb028)
    /usr/local/go/src/net/tcpsock_posix.go:58 +0xb9
net.dialTCP(0x1ed04c0, 0xc4200105a0, 0x15c3bc4, 0x3, 0x0, 0xc4201aa7b0, 0x0, 0x0, 0xf1)
    /usr/local/go/src/net/tcpsock_posix.go:54 +0xe4
net.dialSingle(0x1ed04c0, 0xc4200105a0, 0xc420a46000, 0x1ec7c80, 0xc4201aa7b0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dial.go:501 +0x479
net.dialSerial(0x1ed04c0, 0xc4200105a0, 0xc420a46000, 0xc420a42020, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dial.go:469 +0x223
net.(*Dialer).DialContext(0xc4204eb398, 0x1ed04c0, 0xc4200105a0, 0x15c3bc4, 0x3, 0xc420a42000, 0xf, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/net/dial.go:351 +0x657
net.(*Dialer).Dial(0xc4204eb398, 0x15c3bc4, 0x3, 0xc420a42000, 0xf, 0x0, 0xc4204eb3c0, 0x3, 0x3)
    /usr/local/go/src/net/dial.go:282 +0x75
net.Dial(0x15c3bc4, 0x3, 0xc420a42000, 0xf, 0xf5c2c9, 0xc42034bd00, 0x9, 0x5fc7d0)
    /usr/local/go/src/net/dial.go:261 +0x90
github.com/cockroachdb/pq.defaultDialer.Dial(0x15c3bc4, 0x3, 0xc420a42000, 0xf, 0x8, 0x47000000006006be, 0xc420333550, 0x8)
    /go/src/github.com/cockroachdb/pq/conn.go:93 +0x49
github.com/cockroachdb/pq.(*defaultDialer).Dial(0x223f610, 0x15c3bc4, 0x3, 0xc420a42000, 0xf, 0x15d1667, 0x11, 0x2240620, 0xc420b06010)
    <autogenerated>:2 +0x74
github.com/cockroachdb/pq.dial(0x1ec8e40, 0x223f610, 0xc420369aa0, 0x0, 0xc420333528, 0x0, 0xc420258120)
    /go/src/github.com/cockroachdb/pq/conn.go:371 +0x3fd
github.com/cockroachdb/pq.DialOpen(0x1ec8e40, 0x223f610, 0xc4202e60c0, 0xbc, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/pq/conn.go:330 +0x685
github.com/cockroachdb/pq.Open(0xc4202e60c0, 0xbc, 0xc4204eb8b0, 0xc4202ea9f0, 0xc4201ca348, 0xc42012e660)
    /go/src/github.com/cockroachdb/pq/conn.go:246 +0x70
github.com/cockroachdb/pq.(*drv).Open(0x223f610, 0xc4202e60c0, 0xbc, 0xc4204eb938, 0xc4202ebb00, 0xc42012e680, 0x8)
    /go/src/github.com/cockroachdb/pq/conn.go:45 +0x35
database/sql.(*DB).conn(0xc4201540b0, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/database/sql/sql.go:824 +0x24e
database/sql.(*DB).exec(0xc4201540b0, 0x15d1106, 0x11, 0x0, 0x0, 0x0, 0x21f2a01, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/database/sql/sql.go:1022 +0x97
database/sql.(*DB).Exec(0xc4201540b0, 0x15d1106, 0x11, 0x0, 0x0, 0x0, 0x61a9b3, 0xc4204ebbf0, 0x66e44f, 0xc4204ebc30)
    /usr/local/go/src/database/sql/sql.go:1010 +0xa2
github.com/cockroachdb/cockroach/testutils/sqlutils.(*SQLRunner).Exec(0xc4204ebeb0, 0x15d1106, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/testutils/sqlutils/sql_runner.go:38 +0x85
github.com/cockroachdb/cockroach/testutils/testcluster.TestManualReplication(0xc4201e4180)
    /go/src/github.com/cockroachdb/cockroach/testutils/testcluster/testcluster_test.go:54 +0x26f
testing.tRunner(0xc4201e4180, 0x16ee400)
    /usr/local/go/src/testing/testing.go:610 +0x81
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:646 +0x2ec

goroutine 9 [chan receive]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc4201d2540)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x44
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0x6eb

goroutine 10 [chan receive]:
github.com/cockroachdb/cockroach/rpc.NewContext.func1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:115 +0x66
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420295c00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 11 [select]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:308 +0x3da
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296360)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 12 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc4201d3b00)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xaa3
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x2a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4200f2770)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 13 [select]:
github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/lease.go:1146 +0xcc1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420256570)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 14 [select]:
github.com/cockroachdb/cockroach/sql.NewExecutor.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/executor.go:255 +0x175
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4200144e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 15 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296660)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 16 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202966c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 66 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func2()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:336 +0x5f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296820)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 67 [IO wait]:
net.runtime_pollWait(0x7fef430cf140, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420065b80, 0x72, 0xc4204fd608, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420065b80, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc420065b20, 0x0, 0x1ec2f40, 0xc4206a4020)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc4201ac438, 0xc4204fd6f0, 0xc4204fd700, 0x645340)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc4201ac438, 0x16ecf08, 0xc4202e3cc0, 0xc4201b00b0, 0x0)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4202e3cc0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func3()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:348 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296880)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 68 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296840, 0xc420254000, 0x1487040, 0x1eaa780, 0x15050a0)
    <autogenerated>:9 +0x7f
net/http.(*Server).Serve(0xc4201b6f00, 0x1ecf4c0, 0xc420296840, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:352 +0x3c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202968a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 69 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296860, 0xc4201ac438, 0xc4202540c0, 0x1487040, 0x1eaa780)
    <autogenerated>:9 +0x7f
crypto/tls.(*listener).Accept(0xc4202968c0, 0xc420254060, 0x1487040, 0x1eaa780, 0x15050a0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc4201b6e80, 0x1ec9c80, 0xc4202968c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func5()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:359 +0x42
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202968e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 70 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func6()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:363 +0x6d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296900)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 71 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296780, 0x16ef100, 0xc42016c0a0, 0x1ed6cc0, 0xc42020c000)
    <autogenerated>:9 +0x7f
google.golang.org/grpc.(*Server).Serve(0xc42016c0a0, 0x1ecf4c0, 0xc420296780, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x15a
github.com/cockroachdb/cockroach/server.(*Server).Start.func7()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:370 +0x43
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296920)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 72 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296760, 0xc4201b00e0, 0x0, 0xc420282800, 0x0)
    <autogenerated>:9 +0x7f
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc4201ac418, 0xc42027e090, 0x1ecf4c0, 0xc420296760, 0xc4201b00e0, 0xc420031f98, 0x645340)
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:117 +0x5d
github.com/cockroachdb/cockroach/server.(*Server).Start.func8()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:378 +0xb5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202570e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 73 [chan receive]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:323 +0x65
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296980)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 74 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:863 +0x2af
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4200f2eb0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 75 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:898 +0x2fa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4200f2ec0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 76 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420296be0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 88 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2554 +0x8a6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5180)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 86 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e4c00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 98 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:578 +0x465
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420253950)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 146 [chan receive]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc4206dc9c0)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x44
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0x6eb

goroutine 90 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1134 +0x165
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e51e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 91 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1208 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5200)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 92 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1227 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5220)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 119 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5ba0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 131 [IO wait]:
net.runtime_pollWait(0x7fef430cefc0, 0x72, 0x7)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42026c1b0, 0x72, 0xc420375588, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42026c1b0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc42026c150, 0xc42018c400, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201ac668, 0xc42018c400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4200bef20, 0xc42018c400, 0x400, 0x400, 0x15429a0, 0x0, 0x7fef42fc9000)
    /go/src/github.com/cockroachdb/cmux/cmux.go:218 +0x113
crypto/tls.(*block).readFromUntil(0xc4206e4210, 0x7fef42fc9000, 0xc4200bef20, 0x5, 0xc4200bef20, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc420118700, 0x16efc17, 0xc420118800, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc420118700, 0xc420456000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc42017aa80)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc42017aa80, 0xc420130878, 0x9, 0x9, 0x631270, 0xc4202acae0, 0x0)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc42017aa80, 0xc420130878, 0x9, 0x9, 0x9, 0xc420375ad8, 0x645440, 0xc420328140)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc42017aa80, 0xc420130878, 0x9, 0x9, 0xc4202acae0, 0x0, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc420130878, 0x9, 0x9, 0x1ebe4c0, 0xc42017aa80, 0x0, 0xc400000000, 0xc420375c70, 0xeaaa19)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420130840, 0x20, 0x20, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4206e47b0, 0xc4203e04e0, 0xc4203e04e0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4208fc120, 0xc4206e4840)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x200
google.golang.org/grpc.(*Server).serveStreams(0xc42016c0a0, 0x1ed6a80, 0xc4208fc120)
    /go/src/google.golang.org/grpc/server.go:421 +0x160
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42016c0a0, 0x1ed6c60, 0xc420118700, 0x1ec4cc0, 0xc4200be790)
    /go/src/google.golang.org/grpc/server.go:408 +0x3da
google.golang.org/grpc.(*Server).handleRawConn(0xc42016c0a0, 0x1ed6cc0, 0xc4200bef20)
    /go/src/google.golang.org/grpc/server.go:385 +0x422
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x1a9

goroutine 96 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5a80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 65 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:618 +0x256
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202afe00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 165 [IO wait]:
net.runtime_pollWait(0x7fef430cf080, 0x72, 0x6)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42026c140, 0x72, 0xc4208bd828, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42026c140, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc42026c0e0, 0xc420283800, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420180000, 0xc420283800, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc420013ef0, 0x7fef430d4c08, 0xc420180000, 0x5, 0xc420180000, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc420118000, 0x16efc17, 0xc420118100, 0xc4202ace70)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc420118000, 0xc420468000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc42069f0e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc42069f0e0, 0xc4201e4578, 0x9, 0x9, 0xc4208bdca8, 0x645440, 0xc4203ac080)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc42069f0e0, 0xc4201e4578, 0x9, 0x9, 0x9, 0xc4203e01c0, 0xc4203e01b8, 0x1ec2180)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc42069f0e0, 0xc4201e4578, 0x9, 0x9, 0xc4208bdeb0, 0xea66d0, 0xc42047a000)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4201e4578, 0x9, 0x9, 0x1ebe4c0, 0xc42069f0e0, 0x0, 0xc400000000, 0xc4202ace40, 0xc)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4201e4540, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4202556b0, 0xc4203e0750, 0xc4203e0750, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Client).reader(0xc420350d00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:890 +0xb6
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:172 +0x89c

goroutine 162 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4200f6cc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 97 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5aa0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 114 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5ac0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 115 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5ae0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 116 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5b00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 117 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5b20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 118 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc420064620, 0xecf38bb47, 0xc438b581a6, 0x21f0cc0, 0xc4202e2bc0, 0xc42027e090, 0xc42021a480, 0xc4202f8ec0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc42021a480, 0xc4202eb440)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:224 +0x7c
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc4202eb4d0, 0xc4201dff40)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:274 +0x23e
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:226 +0x1e3
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5b40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 100 [select]:
github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:565 +0x16b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202afee0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 101 [select]:
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:81 +0x16f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202ac480)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 102 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startWriteSummaries.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:652 +0x2bb
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202aff00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 103 [select]:
github.com/cockroachdb/cockroach/sql.(*SchemaChangeManager).Start.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:682 +0x12b7
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202aff20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 120 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc420064a80, 0xecf38bb47, 0xc438b5d2ae, 0x21f0cc0, 0xc4202e2bc0, 0xc42027e090, 0xc42021a480, 0xc4202f8ec0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc42021a480, 0xc4202eb440)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:224 +0x7c
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc4202eb710, 0xc4201dffc0)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:274 +0x23e
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:226 +0x1e3
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202e5bc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 104 [IO wait]:
net.runtime_pollWait(0x7fef430cf200, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420065b10, 0x72, 0xc420d15df8, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420065b10, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc420065ab0, 0x0, 0x1ec2f40, 0xc420912040)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc4201ac420, 0x624bce, 0xc420d15ea8, 0xd438eb)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc4201ac420, 0x16ecf10, 0xc4202e3c80, 0x1ed6f00, 0xc4201800b0)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4202e3c80, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func11()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:439 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc4202affc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 107 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc4201d2e40, 0xc42000ab00, 0xc42027f290, 0xc4200f2e10, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:293 +0x54e
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:189 +0x7f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc420253e30)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 106 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc42016e000)
    /go/src/google.golang.org/grpc/clientconn.go:700 +0x63e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc42016e000)
    /go/src/google.golang.org/grpc/clientconn.go:461 +0x1de
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:462 +0x3c7

goroutine 108 [chan receive]:
database/sql.(*DB).connectionOpener(0xc4201540b0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 129 [select]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc420102270, 0x7fef430505e0, 0xc42033e840, 0xc4202d7760, 0x1ebeb80, 0xc42010e058, 0xc42027fc20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x57d
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x25b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4202e2600)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 164 [IO wait]:
net.runtime_pollWait(0x7fef430ced80, 0x72, 0xb)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200647d0, 0x72, 0xc4209d1598, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200647d0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc420064770, 0xc4203cc000, 0x4000, 0x4000, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42049e048, 0xc4203cc000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4203c4580, 0xc4203cc000, 0x4000, 0x4000, 0x15429a0, 0x0, 0x7fef42fc9000)
    /go/src/github.com/cockroachdb/cmux/cmux.go:218 +0x113
crypto/tls.(*block).readFromUntil(0xc420013e60, 0x7fef42fc9000, 0xc4203c4580, 0x5, 0xc4203c4580, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc4200eb880, 0x16efc17, 0xc4200eb980, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc4200eb880, 0xc42037a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc420848ae0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc420848ae0, 0xc4203ae338, 0x9, 0x9, 0x631270, 0xc42034b090, 0x0)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc420848ae0, 0xc4203ae338, 0x9, 0x9, 0x9, 0xc4209d1ae8, 0x645440, 0xc42017f100)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc420848ae0, 0xc4203ae338, 0x9, 0x9, 0xc42034b090, 0x0, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4203ae338, 0x9, 0x9, 0x1ebe4c0, 0xc420848ae0, 0x0, 0xc400000000, 0xc4209d1c80, 0xeaaa19)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203ae300, 0x1cb, 0x1cb, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc420252990, 0xc42071b1d0, 0xc42071b1d0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4203d42d0, 0xc420252a50)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x200
google.golang.org/grpc.(*Server).serveStreams(0xc42016c0a0, 0x1ed6a80, 0xc4203d42d0)
    /go/src/google.golang.org/grpc/server.go:421 +0x160
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42016c0a0, 0x1ed6c60, 0xc4200eb880, 0x1ec4cc0, 0xc420154420)
    /go/src/google.golang.org/grpc/server.go:408 +0x3da
google.golang.org/grpc.(*Server).handleRawConn(0xc42016c0a0, 0x1ed6cc0, 0xc4203c4580)
    /go/src/google.golang.org/grpc/server.go:385 +0x422
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x1a9

goroutine 38 [chan receive]:
github.com/cockroachdb/cockroach/rpc.NewContext.func1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:115 +0x66
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420153be0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 39 [select]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:308 +0x3da
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc42027c320)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 40 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc4204e2d20)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xaa3
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x2a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206df650)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 41 [select]:
github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/lease.go:1146 +0xcc1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420707290)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 42 [select]:
github.com/cockroachdb/cockroach/sql.NewExecutor.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/executor.go:255 +0x175
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4204e34a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 21 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4202961c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 22 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296280)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 23 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func2()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:336 +0x5f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4202967c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 24 [IO wait]:
net.runtime_pollWait(0x7fef430cee40, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42026c0d0, 0x72, 0xc420710608, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42026c0d0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc42026c070, 0x0, 0x1ec2f40, 0xc4206da000)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420038090, 0xc4207106f0, 0xc420710700, 0x645340)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc420038090, 0x16ecf08, 0xc420016a00, 0xc4202ac030, 0x0)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc420016a00, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func3()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:348 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296aa0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 25 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296800, 0xc4203e0000, 0x1487040, 0x1eaa780, 0x15050a0)
    <autogenerated>:9 +0x7f
net/http.(*Server).Serve(0xc4200a0280, 0x1ecf4c0, 0xc420296800, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:352 +0x3c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296ae0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 26 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296a20, 0xc420038090, 0xc4203e0090, 0x1487040, 0x1eaa780)
    <autogenerated>:9 +0x7f
crypto/tls.(*listener).Accept(0xc420296b00, 0xc4203e0060, 0x1487040, 0x1eaa780, 0x15050a0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc4200a0100, 0x1ec9c80, 0xc420296b00, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func5()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:359 +0x42
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296b20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 27 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func6()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:363 +0x6d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296b80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 28 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4202963c0, 0x16ef100, 0xc420213400, 0x1ed6cc0, 0xc420154630)
    <autogenerated>:9 +0x7f
google.golang.org/grpc.(*Server).Serve(0xc420213400, 0x1ecf4c0, 0xc4202963c0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x15a
github.com/cockroachdb/cockroach/server.(*Server).Start.func7()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:370 +0x43
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296ba0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 29 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420296380, 0xc4202ac060, 0x0, 0xc42003b400, 0x0)
    <autogenerated>:9 +0x7f
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc420038070, 0xc42027fc20, 0x1ecf4c0, 0xc420296380, 0xc4202ac060, 0xc42070b798, 0x645340)
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:117 +0x5d
github.com/cockroachdb/cockroach/server.(*Server).Start.func8()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:378 +0xb5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420852990)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 30 [chan receive]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:323 +0x65
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420296c40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 31 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:863 +0x2af
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420246d60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 32 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:898 +0x2fa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420247140)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 149 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc4204e20c0, 0xc420196100, 0xc42034c1b0, 0xc4202acde0, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:293 +0x54e
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:189 +0x7f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4203e0120)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 148 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc4200ac3c0)
    /go/src/google.golang.org/grpc/clientconn.go:700 +0x63e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4200ac3c0)
    /go/src/google.golang.org/grpc/clientconn.go:461 +0x1de
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:462 +0x3c7

goroutine 110 [IO wait]:
net.runtime_pollWait(0x7fef430cecc0, 0x72, 0xa)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4201c2140, 0x72, 0xc4208ba828, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4201c2140, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4201c20e0, 0xc420410000, 0x1000, 0x1000, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420180040, 0xc420410000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc42033c150, 0x7fef430d4c08, 0xc420180040, 0x5, 0xc420180040, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc420081880, 0x16efc17, 0xc420081980, 0xc4203824e0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc420081880, 0xc4204ca000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc420848de0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc420848de0, 0xc4203ae3f8, 0x9, 0x9, 0x645440, 0xc4201b4e40, 0xc4208bad08)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc420848de0, 0xc4203ae3f8, 0x9, 0x9, 0x9, 0xc420252cf8, 0x1ec2180, 0xc4201d4540)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc420848de0, 0xc4203ae3f8, 0x9, 0x9, 0xea5668, 0xc4204a8960, 0xc4200fa100)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4203ae3f8, 0x9, 0x9, 0x1ebe4c0, 0xc420848de0, 0x0, 0xc400000000, 0x0, 0x100)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203ae3c0, 0x1ec7800, 0xc4201d4450, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc420252b40, 0xc4201d4450, 0xc4201d4450, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Client).reader(0xc4203b0e00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:890 +0xb6
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:172 +0x89c

goroutine 109 [select]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc4203d42d0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x69c
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x767

goroutine 111 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc4203b0e00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:968 +0x59d
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:202 +0xe5d

goroutine 112 [select]:
google.golang.org/grpc.NewClientStream.func1(0x1ed6a20, 0xc4203b0e00, 0xc4204a8960, 0xc42025ea20)
    /go/src/google.golang.org/grpc/stream.go:187 +0x426
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0xa4f

goroutine 113 [runnable]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42084f500, 0xc420335470, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:141 +0x69a
google.golang.org/grpc/transport.(*Stream).Read(0xc4204a8960, 0xc420335470, 0x5, 0x5, 0xc4202fdb00, 0xc4202fdcd0, 0x70dbbb)
    /go/src/google.golang.org/grpc/transport/transport.go:314 +0x59
io.ReadAtLeast(0x1ec2040, 0xc4204a8960, 0xc420335470, 0x5, 0x5, 0x5, 0xc420251118, 0x0, 0x0)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ec2040, 0xc4204a8960, 0xc420335470, 0x5, 0x5, 0xc4201cb868, 0xc4203824e0, 0x0)
    /usr/local/go/src/io/io.go:325 +0x58
google.golang.org/grpc.(*parser).recvMsg(0xc420335460, 0x7fffffff, 0x8b, 0x1, 0x8b, 0xc4200f5540, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0x6f
google.golang.org/grpc.recv(0xc420335460, 0x1ecfd00, 0x223f610, 0xc4204a8960, 0x0, 0x0, 0x1534060, 0xc42018ad20, 0x7fffffff, 0xc420282800, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x4d
google.golang.org/grpc.(*clientStream).RecvMsg(0xc42025ea20, 0x1534060, 0xc42018ad20, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xc4
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc4201b0f70, 0xc4202d7760, 0xc4200f5540, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x62
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x1ed6d20, 0xc4201b0f70, 0xc420102270, 0xc4202d7760, 0xc4204c8f88, 0xc4204c8f90)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x35
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x52
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc420252e40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 178 [runnable]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc4201ca340, 0x1ed6660, 0xc4201b11d0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:157 +0x543
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x14f2f80, 0xc4201ca340, 0x1ed2ee0, 0xc4203d43f0, 0xc4204c8780, 0xc4204c8750)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xbb
google.golang.org/grpc.(*Server).processStreamingRPC(0xc42016c0a0, 0x1ed6a80, 0xc4203d42d0, 0xc4204a8b40, 0xc42026eb70, 0x1ead660, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x702
google.golang.org/grpc.(*Server).handleStream(0xc42016c0a0, 0x1ed6a80, 0xc4203d42d0, 0xc4204a8b40, 0x0)
    /go/src/google.golang.org/grpc/server.go:770 +0xc36
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4201b0ec0, 0xc42016c0a0, 0x1ed6a80, 0xc4203d42d0, 0xc4204a8b40)
    /go/src/google.golang.org/grpc/server.go:419 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0xa3

goroutine 180 [select]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42084f700, 0xc420335870, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:141 +0x69a
google.golang.org/grpc/transport.(*Stream).Read(0xc4204a8b40, 0xc420335870, 0x5, 0x5, 0xc420361968, 0x630f4b, 0x30)
    /go/src/google.golang.org/grpc/transport/transport.go:314 +0x59
io.ReadAtLeast(0x1ec2040, 0xc4204a8b40, 0xc420335870, 0x5, 0x5, 0x5, 0xc42071a120, 0x0, 0xc42071a150)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ec2040, 0xc4204a8b40, 0xc420335870, 0x5, 0x5, 0xc400000001, 0x0, 0x5ff1b9)
    /usr/local/go/src/io/io.go:325 +0x58
google.golang.org/grpc.(*parser).recvMsg(0xc420335860, 0x7fffffff, 0x0, 0x38, 0x16e9b10, 0x16e9b0f, 0xc41ffe54fb, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0x6f
google.golang.org/grpc.recv(0xc420335860, 0x1ecfd00, 0x223f610, 0xc4204a8b40, 0x0, 0x0, 0x1542c40, 0xc420356040, 0x7fffffff, 0x20001, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x4d
google.golang.org/grpc.(*serverStream).RecvMsg(0xc4203d43f0, 0x1542c40, 0xc420356040, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xec
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc4201b11d0, 0x0, 0x0, 0x100010000)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x62
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc4201ca348, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:123 +0x2f
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc4201ca340, 0xc4201ac360, 0xc42084f900, 0xc420361f40, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:267 +0x39f
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:123 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc42084f940)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 173 [select]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc4203d4900)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x69c
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x767

goroutine 324 [IO wait]:
net.runtime_pollWait(0x7fef430cec00, 0x72, 0xc)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200fd330, 0x72, 0xc420962828, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200fd330, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4200fd2d0, 0xc4208f8000, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4200380b0, 0xc4208f8000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc42071a000, 0x7fef430d4c08, 0xc4200380b0, 0x5, 0xc4200380b0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc42097c000, 0x16efc17, 0xc42097c100, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc42097c000, 0xc4209f8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc42086a300)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc42086a300, 0xc4203ae0f8, 0x9, 0x9, 0x9, 0x18, 0x28)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc42086a300, 0xc4203ae0f8, 0x9, 0x9, 0x9, 0x600ee8, 0x30, 0xc420962d18)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc42086a300, 0xc4203ae0f8, 0x9, 0x9, 0x29b54771a5ce8101, 0x2900000000000000, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4203ae0f8, 0x9, 0x9, 0x1ebe4c0, 0xc42086a300, 0x0, 0xc400000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203ae0c0, 0x1ec7a40, 0xc420853350, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc420852960, 0xc420853350, 0xc420853350, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Client).reader(0xc420196900)
    /go/src/google.golang.org/grpc/transport/http2_client.go:890 +0xb6
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:172 +0x89c

goroutine 247 [runnable]:
sync.runtime_Semacquire(0xc4201b6ddc)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc4201b6dd8)
    /usr/local/go/src/sync/mutex.go:85 +0xd0
github.com/cockroachdb/cockroach/gossip.(*infoStore).runCallbacks.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:278 +0x3e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc42027e090, 0x1badc9b, 0x13, 0x122, 0xc4201b13e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc

goroutine 191 [IO wait]:
net.runtime_pollWait(0x7fef430ceb40, 0x72, 0xd)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200fd3a0, 0x72, 0xc420be9588, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200fd3a0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4200fd340, 0xc4208f8400, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201ac4c8, 0xc4208f8400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc420154630, 0xc4208f8400, 0x400, 0x400, 0x15429a0, 0x0, 0x7fef42fc9000)
    /go/src/github.com/cockroachdb/cmux/cmux.go:218 +0x113
crypto/tls.(*block).readFromUntil(0xc420251f50, 0x7fef42fc9000, 0xc420154630, 0x5, 0xc420154630, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc42033a700, 0x16efc17, 0xc42033a800, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc42033a700, 0xc420bb2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc4207c65a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc4207c65a0, 0xc4209b61b8, 0x9, 0x9, 0x9, 0x18, 0x28)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc4207c65a0, 0xc4209b61b8, 0x9, 0x9, 0x9, 0x600ee8, 0x30, 0xc420be9ad8)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc4207c65a0, 0xc4209b61b8, 0x9, 0x9, 0x29b54771a5ce8101, 0x2900000000000000, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4209b61b8, 0x9, 0x9, 0x1ebe4c0, 0xc4207c65a0, 0x0, 0xc400000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4209b6180, 0x1ec7a40, 0xc420853080, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4203dc5d0, 0xc420853080, 0xc420853080, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4203d41b0, 0xc4203dc660)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x200
google.golang.org/grpc.(*Server).serveStreams(0xc420213400, 0x1ed6a80, 0xc4203d41b0)
    /go/src/google.golang.org/grpc/server.go:421 +0x160
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc420213400, 0x1ed6c60, 0xc42033a700, 0x1ec4cc0, 0xc420101810)
    /go/src/google.golang.org/grpc/server.go:408 +0x3da
google.golang.org/grpc.(*Server).handleRawConn(0xc420213400, 0x1ed6cc0, 0xc420154630)
    /go/src/google.golang.org/grpc/server.go:385 +0x422
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x1a9

goroutine 244 [select]:
google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake(0xc42010e050, 0x7fef430d4ba0, 0xc4204b92c0, 0xc42034a6a0, 0xf, 0x1ed6f00, 0xc4201ac678, 0x0, 0x0, 0x40, ...)
    /go/src/google.golang.org/grpc/credentials/credentials.go:154 +0x366
google.golang.org/grpc/transport.newHTTP2Client(0x7fef430d4ba0, 0xc4204b92c0, 0xc42034a6a0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/google.golang.org/grpc/transport/http2_client.go:129 +0x1386
google.golang.org/grpc/transport.NewClientTransport(0x7fef430d4ba0, 0xc4204b92c0, 0xc42034a6a0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/google.golang.org/grpc/transport/transport.go:368 +0x87
google.golang.org/grpc.(*addrConn).resetTransport(0xc4203c2140, 0xc42012e100, 0xc4204c6fa0, 0x8b8e7d)
    /go/src/google.golang.org/grpc/clientconn.go:639 +0x25e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4203c2140)
    /go/src/google.golang.org/grpc/clientconn.go:453 +0x34
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:462 +0x3c7

goroutine 140 [runnable]:
sync.(*Pool).Put(0x21efd70, 0x15b47c0, 0xc420ae2fc0)
    /usr/local/go/src/sync/pool.go:65
math/big.putNat(0xc4203b2ea0, 0x1d, 0x24)
    /usr/local/go/src/math/big/nat.go:555 +0x82
math/big.nat.divLarge(0xc420254420, 0x1, 0x5, 0xc42040a360, 0x1c, 0x24, 0xc42040a480, 0x1c, 0x24, 0xc42040a7e0, ...)
    /usr/local/go/src/math/big/nat.go:634 +0x645
math/big.nat.div(0xc420254420, 0x1, 0x5, 0xc42040a360, 0x1c, 0x24, 0xc42040a480, 0x1c, 0x24, 0xc42040a7e0, ...)
    /usr/local/go/src/math/big/nat.go:541 +0x3cb
math/big.(*Int).QuoRem(0xc420c99208, 0xc420c992a8, 0xc420c991c8, 0xc420c992c8, 0xc420c99208, 0xc420c991c8)
    /usr/local/go/src/math/big/int.go:227 +0xbf
math/big.(*Int).GCD(0xc420c99350, 0xc420700280, 0xc420c99330, 0xc420700260, 0xc4209443e0, 0x100)
    /usr/local/go/src/math/big/int.go:464 +0x435
crypto/rsa.modInverse(0xc420700260, 0xc4209443e0, 0x1, 0xc420700260)
    /usr/local/go/src/crypto/rsa/rsa.go:411 +0xb5
crypto/rsa.decrypt(0x1ebe740, 0xc420012330, 0xc4204b84e0, 0xc420c995b0, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/rsa/rsa.go:490 +0x146
crypto/rsa.decryptAndCheck(0x1ebe740, 0xc420012330, 0xc4204b84e0, 0xc420c995b0, 0xc420c995b0, 0x100, 0x0)
    /usr/local/go/src/crypto/rsa/rsa.go:542 +0x56
crypto/rsa.SignPKCS1v15(0x1ebe740, 0xc420012330, 0xc4204b84e0, 0x5, 0xc420700240, 0x20, 0x20, 0xc420c99648, 0x5fdd1f, 0x14c93e0, ...)
    /usr/local/go/src/crypto/rsa/pkcs1v15.go:250 +0x2b1
crypto/rsa.(*PrivateKey).Sign(0xc4204b84e0, 0x1ebe740, 0xc420012330, 0xc420700240, 0x20, 0x20, 0x1ec3340, 0xc420998070, 0x0, 0x0, ...)
    /usr/local/go/src/crypto/rsa/rsa.go:102 +0x167
crypto/tls.(*ecdheKeyAgreement).generateServerKeyExchange(0xc42033ea40, 0xc4209ba580, 0xc4204b8540, 0xc4204e0000, 0xc4204ee000, 0x34f, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/key_agreement.go:255 +0x67c
crypto/tls.(*serverHandshakeState).doFullHandshake(0xc420c99b28, 0xc4209e4b00, 0x0)
    /usr/local/go/src/crypto/tls/handshake_server.go:397 +0x3de
crypto/tls.(*Conn).serverHandshake(0xc42030a000, 0x16efcd8, 0xc42030a100)
    /usr/local/go/src/crypto/tls/handshake_server.go:86 +0x30d
crypto/tls.(*Conn).Handshake(0xc42030a000, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1234 +0x14e
google.golang.org/grpc/credentials.(*tlsCreds).ServerHandshake(0xc4201ac4a0, 0x1ed6cc0, 0xc420702000, 0x1, 0x1, 0x1ecf140, 0xc42049e080, 0xc420015ce0, 0xc42049e088)
    /go/src/google.golang.org/grpc/credentials/credentials.go:170 +0xd3
google.golang.org/grpc.(*Server).useTransportAuthenticator(0xc4209ad5e0, 0x1ed6cc0, 0xc420702000, 0xc4206defc0, 0x0, 0xc42093a778, 0x61a9b3, 0xc42093a748, 0x8b8e6f)
    /go/src/google.golang.org/grpc/server.go:321 +0x50
google.golang.org/grpc.(*Server).handleRawConn(0xc4209ad5e0, 0x1ed6cc0, 0xc420702000)
    /go/src/google.golang.org/grpc/server.go:364 +0x5a
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x1a9

goroutine 43 [select]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc4208fc120)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x69c
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x767

goroutine 166 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc420350d00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:968 +0x59d
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:202 +0xe5d

goroutine 291 [runnable]:
golang.org/x/net/http2/hpack.appendIndexed(0xc4203b60a0, 0x0, 0x20, 0x3, 0x4, 0x0, 0x3)
    /go/src/golang.org/x/net/http2/hpack/encode.go:152
golang.org/x/net/http2/hpack.(*Encoder).WriteField(0xc4207c74a0, 0x15c710f, 0x7, 0x15c3fd9, 0x4, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/hpack/encode.go:63 +0x41b
google.golang.org/grpc/transport.(*http2Client).NewStream(0xc42007df00, 0x7fef430d3a88, 0xc4201e0bd0, 0xc420ae0140, 0xc420a86400, 0x6006be, 0xc42097a588)
    /go/src/google.golang.org/grpc/transport/http2_client.go:342 +0x8cc
google.golang.org/grpc.sendRequest(0x7fef430d4ba0, 0xc4201d2900, 0x1ecfd00, 0x223f610, 0x0, 0x0, 0xc420ae0140, 0x1ed6a20, 0xc42007df00, 0x15a9600, ...)
    /go/src/google.golang.org/grpc/call.go:81 +0xa0
google.golang.org/grpc.Invoke(0x7fef430d4ba0, 0xc4201d2900, 0x15e0e7f, 0x21, 0x15a9600, 0xc420ae80c8, 0x154efe0, 0xc4201d2c60, 0xc42000b600, 0x0, ...)
    /go/src/google.golang.org/grpc/call.go:185 +0x5c7
github.com/cockroachdb/cockroach/roachpb.(*internalClient).Batch(0xc420aea000, 0x7fef430d4ba0, 0xc4201d2900, 0xc420ae80c8, 0x0, 0x0, 0x0, 0xc4209f2070, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/roachpb/api.pb.go:1545 +0xd2
github.com/cockroachdb/cockroach/kv.(*grpcTransport).SendNext.func1(0xc420ae00a0, 0xc420ae80a0, 0xc4201d26c0)
    /go/src/github.com/cockroachdb/cockroach/kv/transport.go:185 +0xfd
created by github.com/cockroachdb/cockroach/kv.(*grpcTransport).SendNext
    /go/src/github.com/cockroachdb/cockroach/kv/transport.go:194 +0x19a

goroutine 213 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe060)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 47 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:618 +0x256
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe800)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 48 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:578 +0x465
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206e5230)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 194 [select]:
github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:565 +0x16b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe8a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 195 [select]:
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:81 +0x16f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206ef0c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 196 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startWriteSummaries.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:652 +0x2bb
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe8c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 197 [select]:
github.com/cockroachdb/cockroach/sql.(*SchemaChangeManager).Start.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:682 +0x12b7
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe8e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 198 [IO wait]:
net.runtime_pollWait(0x7fef430cef00, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42026c060, 0x72, 0xc42093fe08, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42026c060, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc42026c000, 0x0, 0x1ec2f40, 0xc420335fa0)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420038078, 0x624bce, 0xc42093feb8, 0xd438eb)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc420038078, 0x16ecf10, 0xc4200169c0, 0x1ed6f00, 0xc4201ac4c8)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4200169c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func11()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:439 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe920)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 201 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc4204e20c0, 0xc42007d700, 0xc4208fc1b0, 0xc420246c10, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:293 +0x54e
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:189 +0x7f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206e5530)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 200 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc42016f180)
    /go/src/google.golang.org/grpc/clientconn.go:700 +0x63e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc42016f180)
    /go/src/google.golang.org/grpc/clientconn.go:461 +0x1de
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:462 +0x3c7

goroutine 168 [chan receive]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc42017b560)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x44
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0x6eb

goroutine 204 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da340)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 202 [chan receive]:
database/sql.(*DB).connectionOpener(0xc4200be8f0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 205 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da360)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 245 [select]:
google.golang.org/grpc.(*addrConn).wait(0xc4203c2140, 0x7fef430d4ba0, 0xc42017ade0, 0x1, 0x0, 0x48, 0x16e9b03, 0x16e9b02)
    /go/src/google.golang.org/grpc/clientconn.go:782 +0x173
google.golang.org/grpc.(*ClientConn).getTransport(0xc4200fa000, 0x7fef430d4ba0, 0xc42017ade0, 0x0, 0x0, 0xc42018aaf0, 0x7fef43128e10, 0x61b66c, 0xc4209e7ae8)
    /go/src/google.golang.org/grpc/clientconn.go:505 +0x298
google.golang.org/grpc.Invoke(0x7fef430d4ba0, 0xc42017ade0, 0x15dc8ed, 0x1d, 0x1543180, 0xc420433700, 0x1535720, 0xc42012e720, 0xc4200fa000, 0x0, ...)
    /go/src/google.golang.org/grpc/call.go:167 +0x2bf
github.com/cockroachdb/cockroach/rpc.(*heartbeatClient).Ping(0xc42010e0f8, 0x7fef430d4ba0, 0xc42017ade0, 0xc420433700, 0x0, 0x0, 0x0, 0xc420432688, 0xc420432688, 0xc4204c2d28)
    /go/src/github.com/cockroachdb/cockroach/rpc/heartbeat.pb.go:121 +0xd2
github.com/cockroachdb/cockroach/rpc.(*Context).heartbeat(0xc4204b8780, 0x1ebf340, 0xc42010e0f8, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42034a6a0, 0xf, ...)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:307 +0x16c
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc4204b8780, 0xc4200fa000, 0xc42027ef30, 0xc42034a6a0, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:257 +0x1e3
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:189 +0x7f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4202eaed0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 172 [select]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc4203d41b0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x69c
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x767

goroutine 214 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2554 +0x8a6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe0c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 216 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1134 +0x165
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe100)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 217 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1208 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe120)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 218 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1227 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4208fe140)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 259 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da460)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 206 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da380)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 207 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da3a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 208 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da3c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 209 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da3e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 258 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc42026c2a0, 0xecf38bb48, 0xc4028e0a61, 0x21f0cc0, 0xc4201b5240, 0xc42027fc20, 0x0, 0x61a9b3)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:229 +0x2c1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da400)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 246 [chan receive]:
database/sql.(*DB).connectionOpener(0xc4204480b0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 260 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc420064540, 0xecf38bb48, 0xc402900de8, 0x21f0cc0, 0xc4201b5240, 0xc42027fc20, 0x0, 0x61a9b3)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:229 +0x2c1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027fc20, 0xc4206da480)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 152 [chan receive]:
github.com/cockroachdb/cockroach/rpc.NewContext.func1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:115 +0x66
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42032a280)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 153 [select]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:308 +0x3da
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42032a9a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 154 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc4204b93e0)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xaa3
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x2a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4201b0a60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 155 [select]:
github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/lease.go:1146 +0xcc1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42043cde0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 156 [select]:
github.com/cockroachdb/cockroach/sql.NewExecutor.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/executor.go:255 +0x175
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4204b9c20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 158 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e320)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 159 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e380)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 160 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func2()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:336 +0x5f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e4e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 161 [IO wait]:
net.runtime_pollWait(0x7fef430ce9c0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200fc1b0, 0x72, 0xc420b5e608, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200fc1b0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4200fc150, 0x0, 0x1ec2f40, 0xc42034ea00)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc42049e098, 0xc420b5e6f0, 0xc420b5e700, 0x645340)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc42049e098, 0x16ecf08, 0xc4203563c0, 0xc42034af40, 0x0)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4203563c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func3()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:348 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e560)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 306 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42034e520, 0xc420368540, 0x1487040, 0x1eaa780, 0x15050a0)
    <autogenerated>:9 +0x7f
net/http.(*Server).Serve(0xc4200a0380, 0x1ecf4c0, 0xc42034e520, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:352 +0x3c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e580)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 307 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42034e540, 0xc42049e098, 0xc4203685d0, 0x1487040, 0x1eaa780)
    <autogenerated>:9 +0x7f
crypto/tls.(*listener).Accept(0xc42034e5a0, 0xc4203685a0, 0x1487040, 0x1eaa780, 0x15050a0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc4200a0300, 0x1ec9c80, 0xc42034e5a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/cockroachdb/cockroach/server.(*Server).Start.func5()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:359 +0x42
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e5c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 308 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func6()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:363 +0x6d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e5e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 309 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42034e440, 0x16ef100, 0xc4209ad5e0, 0x1ed6cc0, 0xc420702000)
    <autogenerated>:9 +0x7f
google.golang.org/grpc.(*Server).Serve(0xc4209ad5e0, 0x1ecf4c0, 0xc42034e440, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x15a
github.com/cockroachdb/cockroach/server.(*Server).Start.func7()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:370 +0x43
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e600)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 310 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42034e420, 0xc42034af50, 0x0, 0xc42018c000, 0x0)
    <autogenerated>:9 +0x7f
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc42049e078, 0xc4208fc240, 0x1ecf4c0, 0xc42034e420, 0xc42034af50, 0xc420b60f98, 0x645340)
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:117 +0x5d
github.com/cockroachdb/cockroach/server.(*Server).Start.func8()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:378 +0xb5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203680f0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 311 [chan receive]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:323 +0x65
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e660)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 312 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:863 +0x2af
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034a730)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 313 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:898 +0x2fa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034a740)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 314 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034e980)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 315 [select]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc4201bcdd0, 0x7fef430505e0, 0xc420357300, 0xc4209ba6e0, 0x1ebeb80, 0xc42049e0d8, 0xc4208fc240, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x57d
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x25b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc420357100)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 318 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc4204b8780, 0xc42000b600, 0xc42034c090, 0xc4206ef5f0, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:293 +0x54e
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:189 +0x7f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203687e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 317 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420c96500)
    /go/src/google.golang.org/grpc/clientconn.go:700 +0x63e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420c96500)
    /go/src/google.golang.org/grpc/clientconn.go:461 +0x1de
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:462 +0x3c7

goroutine 174 [IO wait]:
net.runtime_pollWait(0x7fef430ce900, 0x72, 0x10)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200fcc30, 0x72, 0xc42095f828, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200fcc30, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4200fcbd0, 0xc42018cc00, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42049e0e8, 0xc42018cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc4203dc690, 0x7fef430d4c08, 0xc42049e0e8, 0x5, 0xc42049e0e8, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc42033aa80, 0x16efc17, 0xc42033ab80, 0x5fefbc)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc42033aa80, 0xc4203fe000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc4207c7440)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc4207c7440, 0xc4209b6578, 0x9, 0x9, 0x645440, 0xc4203ac300, 0xc42095fd08)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc4207c7440, 0xc4209b6578, 0x9, 0x9, 0x9, 0xc4203dd7a8, 0x1ec2180, 0xc420252a80)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc4207c7440, 0xc4209b6578, 0x9, 0x9, 0xea5668, 0xc4202385a0, 0xc4209de4c0)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4209b6578, 0x9, 0x9, 0x1ebe4c0, 0xc4207c7440, 0x0, 0xc400000000, 0x0, 0x37)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4209b6540, 0x1ec7800, 0xc420252960, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4203dd6e0, 0xc420252960, 0xc420252960, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Client).reader(0xc42007df00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:890 +0xb6
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:172 +0x89c

goroutine 323 [IO wait]:
net.runtime_pollWait(0x7fef430ce840, 0x72, 0x11)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42026c370, 0x72, 0xc420973588, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42026c370, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc42026c310, 0xc42018d400, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201800b0, 0xc42018d400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc42020c000, 0xc42018d400, 0x400, 0x400, 0x15429a0, 0x0, 0x7fef42fc9000)
    /go/src/github.com/cockroachdb/cmux/cmux.go:218 +0x113
crypto/tls.(*block).readFromUntil(0xc4203dd050, 0x7fef42fc9000, 0xc42020c000, 0x5, 0xc42020c000, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc42024dc00, 0x16efc17, 0xc42024dd00, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc42024dc00, 0xc420954000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc4207c7140)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc4207c7140, 0xc4209b64b8, 0x9, 0x9, 0x631270, 0xc420974570, 0x0)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1ebe4c0, 0xc4207c7140, 0xc4209b64b8, 0x9, 0x9, 0x9, 0xc420973ad8, 0x645440, 0xc420016c80)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ebe4c0, 0xc4207c7140, 0xc4209b64b8, 0x9, 0x9, 0xc420974570, 0x0, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc4209b64b8, 0x9, 0x9, 0x1ebe4c0, 0xc4207c7140, 0x0, 0xc400000000, 0xc420973c70, 0xeaaa19)
    /go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4209b6480, 0xeb, 0xeb, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4203dd5c0, 0xc420257a10, 0xc420257a10, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4203d4900, 0xc4203dd650)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x200
google.golang.org/grpc.(*Server).serveStreams(0xc42016c0a0, 0x1ed6a80, 0xc4203d4900)
    /go/src/google.golang.org/grpc/server.go:421 +0x160
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42016c0a0, 0x1ed6c60, 0xc42024dc00, 0x1ec4cc0, 0xc420154a50)
    /go/src/google.golang.org/grpc/server.go:408 +0x3da
google.golang.org/grpc.(*Server).handleRawConn(0xc42016c0a0, 0x1ed6cc0, 0xc42020c000)
    /go/src/google.golang.org/grpc/server.go:385 +0x422
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x1a9

goroutine 325 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc420196900)
    /go/src/google.golang.org/grpc/transport/http2_client.go:968 +0x59d
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:202 +0xe5d

goroutine 175 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc42007df00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:968 +0x59d
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:202 +0xe5d

goroutine 176 [select]:
google.golang.org/grpc.NewClientStream.func1(0x1ed6a20, 0xc42007df00, 0xc4202385a0, 0xc420a00b40)
    /go/src/google.golang.org/grpc/stream.go:187 +0x426
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0xa4f

goroutine 177 [select]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4200f9f00, 0xc4203b60d0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:141 +0x69a
google.golang.org/grpc/transport.(*Stream).Read(0xc4202385a0, 0xc4203b60d0, 0x5, 0x5, 0x61e15b, 0xc420953be8, 0x645550)
    /go/src/google.golang.org/grpc/transport/transport.go:314 +0x59
io.ReadAtLeast(0x1ec2040, 0xc4202385a0, 0xc4203b60d0, 0x5, 0x5, 0x5, 0x3440, 0x188400fc4f, 0xc4200f41e0)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ec2040, 0xc4202385a0, 0xc4203b60d0, 0x5, 0x5, 0xc42007e27c, 0xc420467a00, 0x1)
    /usr/local/go/src/io/io.go:325 +0x58
google.golang.org/grpc.(*parser).recvMsg(0xc4203b60c0, 0x7fffffff, 0x32, 0x0, 0x32, 0xc420ae0000, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0x6f
google.golang.org/grpc.recv(0xc4203b60c0, 0x1ecfd00, 0x223f610, 0xc4202385a0, 0x0, 0x0, 0x1534060, 0xc420ae0050, 0x7fffffff, 0xc420a86400, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x4d
google.golang.org/grpc.(*clientStream).RecvMsg(0xc420a00b40, 0x1534060, 0xc420ae0050, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xc4
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc4200f3f60, 0xc4209ba6e0, 0xc420ae0000, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x62
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x1ed6d20, 0xc4200f3f60, 0xc4201bcdd0, 0xc4209ba6e0, 0xc4204c8f88, 0xc4204c8f90)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x35
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x52
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203dd860)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 338 [runnable]:
sync.(*Mutex).Lock(0xc4201ca348)
    /usr/local/go/src/sync/mutex.go:44
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc4201ca340, 0x1ed6660, 0xc4206ee260, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:132 +0x3a0
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x14f2f80, 0xc4201ca340, 0x1ed2ee0, 0xc4203d4a20, 0xc4204c2f80, 0xc4204c2ef0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xbb
google.golang.org/grpc.(*Server).processStreamingRPC(0xc42016c0a0, 0x1ed6a80, 0xc4203d4900, 0xc420238780, 0xc42026eb70, 0x1ead660, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x702
google.golang.org/grpc.(*Server).handleStream(0xc42016c0a0, 0x1ed6a80, 0xc4203d4900, 0xc420238780, 0x0)
    /go/src/google.golang.org/grpc/server.go:770 +0xc36
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4200f3e90, 0xc42016c0a0, 0x1ed6a80, 0xc4203d4900, 0xc420238780)
    /go/src/google.golang.org/grpc/server.go:419 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0xa3

goroutine 340 [select]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42049c200, 0xc4203b64d0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:141 +0x69a
google.golang.org/grpc/transport.(*Stream).Read(0xc420238780, 0xc4203b64d0, 0x5, 0x5, 0xc420909968, 0x630f4b, 0x40)
    /go/src/google.golang.org/grpc/transport/transport.go:314 +0x59
io.ReadAtLeast(0x1ec2040, 0xc420238780, 0xc4203b64d0, 0x5, 0x5, 0x5, 0xc420433680, 0x0, 0xc4204336c0)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1ec2040, 0xc420238780, 0xc4203b64d0, 0x5, 0x5, 0xc400000001, 0x0, 0x5ff1b9)
    /usr/local/go/src/io/io.go:325 +0x58
google.golang.org/grpc.(*parser).recvMsg(0xc4203b64c0, 0x7fffffff, 0x0, 0x38, 0x16e9b10, 0x16e9b0f, 0xc41ffb10db, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0x6f
google.golang.org/grpc.recv(0xc4203b64c0, 0x1ecfd00, 0x223f610, 0xc420238780, 0x0, 0x0, 0x1542c40, 0xc4209de440, 0x7fffffff, 0x20001, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x4d
google.golang.org/grpc.(*serverStream).RecvMsg(0xc4203d4a20, 0x1542c40, 0xc4209de440, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xec
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc4206ee260, 0x0, 0x0, 0x100010000)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x62
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc4201ca348, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:123 +0x2f
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc4201ca340, 0xc4201ac4c0, 0xc42049c440, 0xc420909f40, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:267 +0x39f
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:123 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc42027e090, 0xc42049c480)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 364 [select]:
github.com/cockroachdb/cockroach/sql.(*SchemaChangeManager).Start.func1()
    /go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:682 +0x12b7
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7c60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 363 [semacquire]:
sync.runtime_Semacquire(0xc4209add6c)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc4209add68)
    /usr/local/go/src/sync/mutex.go:85 +0xd0
github.com/cockroachdb/cockroach/server/status.(*MetricsRecorder).GetStatusSummary(0xc4209add60, 0x0)
    /go/src/github.com/cockroachdb/cockroach/server/status/recorder.go:244 +0x80
github.com/cockroachdb/cockroach/server.(*Node).writeSummaries.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:669 +0x5f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc4208fc240, 0xc4209a7e00, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:215 +0x100
github.com/cockroachdb/cockroach/server.(*Node).writeSummaries(0xc420cbc000, 0xc4204c7670, 0x5f8648)
    /go/src/github.com/cockroachdb/cockroach/server/node.go:690 +0x69
github.com/cockroachdb/cockroach/server.(*Node).startWriteSummaries.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:646 +0x60
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7c40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 362 [runnable]:
github.com/cockroachdb/cockroach/ts.MakeDataKey(0xc4203c94e0, 0x18, 0xc420a7ed98, 0x1, 0x1, 0x14686dcbb3df2000, 0x1, 0x34630b8a000, 0x2540be400)
    /go/src/github.com/cockroachdb/cockroach/ts/keys.go:65
github.com/cockroachdb/cockroach/ts.(*DB).StoreData(0xc42049e010, 0x1, 0xc42025a000, 0xc3, 0x124, 0x8b75f2, 0xc42094edd8)
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:126 +0x339
github.com/cockroachdb/cockroach/ts.(*poller).poll.func1()
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:100 +0x7d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc4208fc240, 0xc42094ee88, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:215 +0x100
github.com/cockroachdb/cockroach/ts.(*poller).poll(0xc42033d500)
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:103 +0x4e
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
    /go/src/github.com/cockroachdb/cockroach/ts/db.go:77 +0x41
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4206deef0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 361 [select]:
github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:565 +0x16b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7c20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 360 [select]:
github.com/cockroachdb/cockroach/kv.(*DistSender).sendToReplicas(0xc4203d4000, 0x7fef430d3a88, 0xc420253080, 0x2540be400, 0xb2d05e00, 0x0, 0x1, 0xc420ca05e0, 0x1, 0x1, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:937 +0xa77
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc4203d4000, 0x7fef430d3a88, 0xc420253080, 0x1, 0xc420ca05e0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:364 +0x1c6
github.com/cockroachdb/cockroach/kv.(*DistSender).sendSingleRange(0xc4203d4000, 0x7fef430d3a88, 0xc420253080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4209b6c00, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:480 +0x19f
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk.func2(0xc4209eea20, 0xc4209ee8c0, 0xc4209ef108, 0xc4203d4000, 0xc4209ef0f8, 0x8, 0xc420974738)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:739 +0x18e
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk(0xc4203d4000, 0x7fef430d3a88, 0xc420253080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4209b6c00, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:740 +0x437
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc4203d4000, 0x7fef430d3a88, 0xc420253080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4209b6c00, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:584 +0x3c0
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc4204b93e0, 0x7fef430d3a88, 0xc420253080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42097a9f8, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:375 +0x546
github.com/cockroachdb/cockroach/internal/client.(*txnSender).Send(0xc42097a9a0, 0x7fef430ce240, 0xc4200105a8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42097a9f8, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:52 +0xb6
github.com/cockroachdb/cockroach/internal/client.(*DB).send(0xc42097a9a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:486 +0x220
github.com/cockroachdb/cockroach/internal/client.(*Txn).send(0xc42097a9a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:675 +0x4d4
github.com/cockroachdb/cockroach/internal/client.(*Txn).(github.com/cockroachdb/cockroach/internal/client.send)-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0x58
github.com/cockroachdb/cockroach/internal/client.sendAndFill(0xc4209f0350, 0xc4203c7200, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:415 +0x16b
github.com/cockroachdb/cockroach/internal/client.(*Txn).Run(0xc42097a9a0, 0xc4203c7200, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0xc6
github.com/cockroachdb/cockroach/internal/client.runOneResult(0x1ebec80, 0xc42097a9a0, 0xc4203c7200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:513 +0x80
github.com/cockroachdb/cockroach/internal/client.runOneRow(0x1ebec80, 0xc42097a9a0, 0xc4203c7200, 0x18, 0xc420ca04e0, 0x0, 0xc420068400, 0x10, 0x7fef43128000)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:527 +0x5a
github.com/cockroachdb/cockroach/internal/client.(*Txn).Get(0xc42097a9a0, 0x1528ca0, 0xc420ca04e0, 0xc420ca04e0, 0x18, 0x18, 0xc420974701, 0xc420ca04e0, 0xc4209f05d8)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:221 +0xbd
github.com/cockroachdb/cockroach/internal/client.(*Txn).GetProto(0xc42097a9a0, 0x1528ca0, 0xc420ca04e0, 0x1ecde00, 0xc420974770, 0x10, 0xc420252ed0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:229 +0x49
github.com/cockroachdb/cockroach/sql.(*planner).getDescriptor(0xc420243000, 0x1ec8a00, 0xc420938600, 0x1ed8020, 0xc4209ae660, 0xc42026fb00, 0x160906d, 0x6)
    /go/src/github.com/cockroachdb/cockroach/sql/descriptor.go:167 +0x209
github.com/cockroachdb/cockroach/sql.(*planner).getDatabaseDesc(0xc420243000, 0x160906d, 0x6, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/sql/database.go:133 +0x22e
github.com/cockroachdb/cockroach/sql.(*planner).mustGetDatabaseDesc(0xc420243000, 0x160906d, 0x6, 0x0, 0x0, 0x149be20)
    /go/src/github.com/cockroachdb/cockroach/sql/database.go:142 +0x43
github.com/cockroachdb/cockroach/sql.(*planner).getTableDesc(0xc420243000, 0xc4203accc0, 0x2, 0x1, 0xc4201e2960)
    /go/src/github.com/cockroachdb/cockroach/sql/table.go:130 +0xe0
github.com/cockroachdb/cockroach/sql.(*planner).mustGetTableDesc(0xc420243000, 0xc4203accc0, 0x160906d, 0x6, 0x2240620)
    /go/src/github.com/cockroachdb/cockroach/sql/table.go:148 +0x39
github.com/cockroachdb/cockroach/sql.(*planner).getTableLease(0xc420243000, 0xc4203accc0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/sql/table.go:178 +0x8eb
github.com/cockroachdb/cockroach/sql.(*planner).getAliasedTableLease(0xc420243000, 0x1ec6d00, 0xc4203acd00, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/sql/table.go:334 +0x83
github.com/cockroachdb/cockroach/sql.(*planner).makeEditNode(0xc420243000, 0x1ec6d00, 0xc4203acd00, 0x600000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/sql/update.go:43 +0x88
github.com/cockroachdb/cockroach/sql.(*planner).Insert(0xc420243000, 0xc420916a50, 0x0, 0x0, 0x0, 0x0, 0xc420938590, 0x0, 0x1449a20, 0xc4209383b0)
    /go/src/github.com/cockroachdb/cockroach/sql/insert.go:58 +0x9e
github.com/cockroachdb/cockroach/sql.(*planner).newPlan(0xc420243000, 0x1ed0e80, 0xc420916a50, 0x0, 0x0, 0x0, 0x1449000, 0xc420938598, 0x0, 0x1ed0e80, ...)
    /go/src/github.com/cockroachdb/cockroach/sql/plan.go:260 +0x16fe
github.com/cockroachdb/cockroach/sql.(*planner).makePlan(0xc420243000, 0x1ed0e80, 0xc420916a50, 0x0, 0x0, 0x0, 0xc42084a000, 0x0)
    /go/src/github.com/cockroachdb/cockroach/sql/plan.go:212 +0x70
github.com/cockroachdb/cockroach/sql.(*planner).query(0xc420243000, 0x1609060, 0x74, 0xc4209f1978, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/sql/planner.go:215 +0xbd
github.com/cockroachdb/cockroach/sql.(*planner).exec(0xc420243000, 0x1609060, 0x74, 0xc4209f1978, 0x5, 0x5, 0x5fdc6b, 0x1449a20, 0xc4209383b0)
    /go/src/github.com/cockroachdb/cockroach/sql/planner.go:243 +0x6d
github.com/cockroachdb/cockroach/sql.InternalExecutor.ExecuteStatementInTransaction(0xc4209adcc0, 0xc42097a9a0, 0x1609060, 0x74, 0xc4209f1978, 0x5, 0x5, 0xc4209a7996, 0xc41ffc91e4, 0xc4209a79a0)
    /go/src/github.com/cockroachdb/cockroach/sql/internal.go:46 +0xb5
github.com/cockroachdb/cockroach/sql.EventLogger.InsertEventRecord(0xc4209adcc0, 0xc42097a9a0, 0x15ca3da, 0x9, 0x300000003, 0x14f8020, 0xc4206dc360, 0x0, 0x3fc3333333333333)
    /go/src/github.com/cockroachdb/cockroach/sql/event_log.go:134 +0x395
github.com/cockroachdb/cockroach/server.(*Node).recordJoinEvent.func1.1(0xc42097a9a0, 0x0, 0x1)
    /go/src/github.com/cockroachdb/cockroach/server/node.go:723 +0x126
github.com/cockroachdb/cockroach/internal/client.(*DB).Txn.func1(0xc42097a9a0, 0xc42034ff40, 0x4000000000000000, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:450 +0x2f
github.com/cockroachdb/cockroach/internal/client.(*Txn).Exec(0xc42097a9a0, 0xc420340101, 0x0, 0x0, 0xc42034bb60, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:553 +0x232
github.com/cockroachdb/cockroach/internal/client.(*DB).Txn(0xc42099b950, 0xc42034ff20, 0x4000000000000000, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:451 +0x28d
github.com/cockroachdb/cockroach/server.(*Node).recordJoinEvent.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:724 +0x1d9
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7ba0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 359 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:578 +0x465
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42033d4d0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 358 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:618 +0x256
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7b80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 357 [runnable]:
sync.runtime_Semacquire(0xc4209add6c)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc4209add68)
    /usr/local/go/src/sync/mutex.go:85 +0xd0
github.com/cockroachdb/cockroach/server/status.(*MetricsRecorder).GetStatusSummary(0xc4209add60, 0x0)
    /go/src/github.com/cockroachdb/cockroach/server/status/recorder.go:244 +0x80
github.com/cockroachdb/cockroach/server.(*Node).writeSummaries.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:669 +0x5f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc4208fc240, 0xc42096fdc8, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:215 +0x100
github.com/cockroachdb/cockroach/server.(*Node).writeSummaries(0xc420cbc000, 0x7fef430ce240, 0xc4200105a0)
    /go/src/github.com/cockroachdb/cockroach/server/node.go:690 +0x69
github.com/cockroachdb/cockroach/server.(*Node).bootstrapStores(0xc420cbc000, 0x7fef430ce240, 0xc4200105a0, 0xc42049e0b8, 0x1, 0x1, 0xc4208fc240)
    /go/src/github.com/cockroachdb/cockroach/server/node.go:531 +0x443
github.com/cockroachdb/cockroach/server.(*Node).initStores.func1()
    /go/src/github.com/cockroachdb/cockroach/server/node.go:461 +0x66
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc4208fc240, 0x1bad4f8, 0xe, 0x1ce, 0xc42049dd00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc

goroutine 374 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2554 +0x8a6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034f940)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 373 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:275 +0x217
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034f8e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 376 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1134 +0x165
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034f9a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 377 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1208 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034f9c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 378 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1227 +0x263
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034f9e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 383 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fc40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 381 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fae0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 382 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc4200fc4d0, 0xecf38bb48, 0xc4059c6cdd, 0x21f0cc0, 0xc420432680, 0xc4208fc240, 0x0, 0x61a9b3)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:229 +0x2c1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fb00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 384 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fc60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 385 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fc80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 386 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fca0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 387 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fcc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 388 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2a1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fce0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 389 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc4200fc1c0, 0xecf38bb48, 0xc4059defc0, 0x21f0cc0, 0xc420432680, 0xc4208fc240, 0x0, 0x61a9b3)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:176 +0x382
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:229 +0x2c1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc42034fd00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 365 [IO wait]:
net.runtime_pollWait(0x7fef430cea80, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4200fc140, 0x72, 0xc4209e3df8, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4200fc140, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4200fc0e0, 0x0, 0x1ec2f40, 0xc420700040)
    /usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc42049e080, 0x624bce, 0xc4209e3ea8, 0xd438eb)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc42049e080, 0x16ecf10, 0xc420356380, 0x1ed6f00, 0xc4208f2008)
    /usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/cockroachdb/cmux.(*cMux).Serve(0xc420356380, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/server.(*Server).Start.func11()
    /go/src/github.com/cockroachdb/cockroach/server/server.go:439 +0x2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc4208fc240, 0xc4203b7ca0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x66

goroutine 367 [IO wait]:
net.runtime_pollWait(0x7fef430ce780, 0x72, 0x12)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4209b1f70, 0x72, 0xc420d12838, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4209b1f70, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4209b1f10, 0xc420983000, 0x400, 0x400, 0x0, 0x1ec5a80, 0xc4200100b8)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201ac678, 0xc420983000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc420257740, 0x7fef430d4c08, 0xc4201ac678, 0x5, 0xc4201ac678, 0xc4209f7a00)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc4207a0380, 0x416, 0x91, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).readHandshake(0xc4207a0380, 0xc420109116, 0xc420109170, 0x8c, 0x8c)
    /usr/local/go/src/crypto/tls/conn.go:925 +0x8f
crypto/tls.(*Conn).clientHandshake(0xc4207a0380, 0x16efcd8, 0xc4207a0480)
    /usr/local/go/src/crypto/tls/handshake_client.go:159 +0x99f
crypto/tls.(*Conn).Handshake(0xc4207a0380, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1232 +0xd3
google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake.func1(0xc420848000, 0xc4207a0380)
    /go/src/google.golang.org/grpc/credentials/credentials.go:152 +0x2b
created by google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake
    /go/src/google.golang.org/grpc/credentials/credentials.go:153 +0x132

goroutine 248 [semacquire]:
sync.runtime_Semacquire(0xc4201b6ddc)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc4201b6dd8)
    /usr/local/go/src/sync/mutex.go:85 +0xd0
github.com/cockroachdb/cockroach/gossip.(*infoStore).runCallbacks.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:278 +0x3e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc42027e090, 0x1badc9b, 0x13, 0x122, 0xc4201b1410)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc

goroutine 249 [semacquire]:
sync.runtime_Semacquire(0xc4201b6ddc)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc4201b6dd8)
    /usr/local/go/src/sync/mutex.go:85 +0xd0
github.com/cockroachdb/cockroach/gossip.(*infoStore).runCallbacks.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:278 +0x3e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc42027e090, 0x1badc9b, 0x13, 0x122, 0xc4201b1420)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc

ERROR: exit status 2

Run Details:

31 runs so far, 0 failures, over 5s
65 runs so far, 0 failures, over 10s
106 runs so far, 0 failures, over 15s
109 runs completed, 1 failures, over 15s
FAIL

Please assign, take a look and update the issue accordingly.

cockroachdb / cockroach

stress: failed test in cockroach/testutils/testcluster/testcluster.test: TestManualReplication #8359

8349.