search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

sql/sqlbase: TestAdjustStartKeyForInterleave failed under stress #30493

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/ee5696d9afdc7cda7f7cea6520928f98af58e335

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestAdjustStartKeyForInterleave PKG=github.com/cockroachdb/cockroach/pkg/sql/sqlbase TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=919014&tab=buildLog


I180921 06:58:57.417633 4909 base/addr_validation.go:319  [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180921 06:58:57.422461 4909 server/config.go:493  [n?] 1 storage engine initialized
I180921 06:58:57.422551 4909 server/config.go:496  [n?] RocksDB cache size: 128 MiB
I180921 06:58:57.422693 4909 server/config.go:496  [n?] store 0: in-memory, size 0 B
I180921 06:58:57.476407 4909 util/stop/stopper.go:537  [n?] quiescing; tasks left:
1      [async] storage.pendingLeaseRequest: requesting lease
I180921 06:58:57.479867 4909 server/node.go:374  [n?] **** cluster 44786663-72ec-46b0-9a5b-94bd5bc51ccf has been created
I180921 06:58:57.479971 4909 server/server.go:1414  [n?] **** add additional nodes by specifying --join=127.0.0.1:33635
I180921 06:58:57.480676 4909 gossip/gossip.go:408  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:33635" > attrs:<> locality:<> ServerVersion:<major_val:2 minor_val:0 patch:0 unstable:12 > build_tag:"v2.1.0-beta.20180917-206-gee5696d" started_at:1537513137480448162 
I180921 06:58:57.484125 5000 gossip/gossip.go:940  [n1] gossip connectivity
I180921 06:58:57.501825 4909 storage/store.go:1559  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180921 06:58:57.509179 4909 server/node.go:477  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=7.0 KiB), ranges=1, leases=1, queries=0.00, writes=0.00, bytesPerReplica={p10=7183.00 p25=7183.00 p50=7183.00 p75=7183.00 p90=7183.00 pMax=7183.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I180921 06:58:57.509463 4909 storage/stores.go:242  [n1] read 0 node addresses from persistent storage
I180921 06:58:57.509872 4909 server/node.go:700  [n1] connecting to gossip network to verify cluster ID...
I180921 06:58:57.510015 4909 server/node.go:725  [n1] node connected via gossip and verified as part of cluster "44786663-72ec-46b0-9a5b-94bd5bc51ccf"
I180921 06:58:57.510520 4909 server/node.go:549  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180921 06:58:57.512495 4909 server/status/recorder.go:611  [n1] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180921 06:58:57.512610 4909 server/server.go:1805  [n1] Could not start heap profiler worker due to: directory to store profiles could not be determined
I180921 06:58:57.518959 4909 server/server.go:1551  [n1] starting https server at 127.0.0.1:42551 (use: 127.0.0.1:42551)
I180921 06:58:57.519071 4909 server/server.go:1553  [n1] starting grpc/postgres server at 127.0.0.1:33635
I180921 06:58:57.519137 4909 server/server.go:1554  [n1] advertising CockroachDB node at 127.0.0.1:33635
I180921 06:58:57.530379 5187 storage/replica_command.go:298  [n1,split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180921 06:58:57.616907 4936 storage/replica_command.go:298  [n1,split,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180921 06:58:57.720763 4896 storage/replica_command.go:298  [n1,split,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
W180921 06:58:57.727283 4894 storage/intent_resolver.go:668  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=e8f88a21 key=/Table/SystemConfigSpan/Start rw=true pri=0.02007936 iso=SERIALIZABLE stat=PENDING epo=0 ts=1537513137.550505655,0 orig=1537513137.550505655,0 max=1537513137.550505655,0 wto=false rop=false seq=6
I180921 06:58:57.736773 5225 sql/event_log.go:126  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I180921 06:58:57.774927 5255 storage/replica_command.go:298  [n1,split,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180921 06:58:57.831410 5016 storage/replica_command.go:298  [n1,split,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180921 06:58:57.889977 5199 storage/replica_command.go:298  [n1,split,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180921 06:58:57.897031 5252 sql/event_log.go:126  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:2.0-12 User:root}
I180921 06:58:57.948677 5284 storage/replica_command.go:298  [n1,split,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180921 06:58:58.029453 5266 sql/event_log.go:126  [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root}
I180921 06:58:58.042293 5301 storage/replica_command.go:298  [n1,split,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180921 06:58:58.094021 5105 storage/replica_command.go:298  [n1,split,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180921 06:58:58.169183 5289 storage/replica_command.go:298  [n1,split,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180921 06:58:58.207296 5295 storage/replica_command.go:298  [n1,split,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180921 06:58:58.209250 5313 sql/event_log.go:126  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:11377741-c2c7-433c-ac19-94365231b0bf User:root}
I180921 06:58:58.247603 5323 storage/replica_command.go:298  [n1,split,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180921 06:58:58.260601 5320 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I180921 06:58:58.292866 5379 storage/replica_command.go:298  [n1,split,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180921 06:58:58.319018 5248 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I180921 06:58:58.341389 4909 server/server.go:1607  [n1] done ensuring all necessary migrations have run
I180921 06:58:58.341589 4909 server/server.go:1610  [n1] serving sql connections
I180921 06:58:58.367493 5367 server/server_update.go:67  [n1] no need to upgrade, cluster already at the newest version
I180921 06:58:58.369012 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.369753 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.370853 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.371563 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.372340 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.372914 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.373499 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.374001 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.374506 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.374997 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.375513 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.376060 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.376114 5369 sql/event_log.go:126  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:33635} Attrs: Locality: ServerVersion:2.0-12 BuildTag:v2.1.0-beta.20180917-206-gee5696d StartedAt:1537513137480448162 LocalityAddress:[]} ClusterID:44786663-72ec-46b0-9a5b-94bd5bc51ccf StartedAt:1537513137480448162 LastUp:1537513137480448162}
I180921 06:58:58.376735 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.377376 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.378002 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.378650 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.379300 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.380003 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.380922 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.381895 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.383097 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.384775 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.387603 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.392679 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.402350 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.419951 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.454425 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.522378 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.629040 5339 rpc/nodedialer/nodedialer.go:92  [n1,consistencyChecker,s1,r1/1:/{Min-System/}] connection to n1 established
I180921 06:58:58.657444 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:58.926758 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:58:59.424237 5000 gossip/gossip.go:940  [n1] gossip connectivity
  n1 [sentinel];
I180921 06:58:59.464571 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:00.466746 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:01.467617 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:02.468598 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:03.469553 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:04.470608 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:05.471546 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:06.472939 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:06.514773 5126 storage/replica_proposal.go:211  [n1,s1,r9/1:/Table/1{2-3}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.512172605,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.527195 5128 storage/replica_proposal.go:211  [n1,s1,r6/1:/{System/tse-Table/System…}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.517528286,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.539487 5127 storage/replica_proposal.go:211  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.536935539,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.545583 5119 storage/replica_proposal.go:211  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.543118909,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.553758 5132 storage/replica_proposal.go:211  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.551181070,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.558972 5141 storage/replica_proposal.go:211  [n1,s1,r13/1:/Table/1{6-7}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.556471232,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.563312 5137 storage/replica_proposal.go:211  [n1,s1,r10/1:/Table/1{3-4}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.560824612,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.569252 5138 storage/replica_proposal.go:211  [n1,s1,r11/1:/Table/1{4-5}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.566106504,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:06.573223 5135 storage/replica_proposal.go:211  [n1,s1,r8/1:/Table/1{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513146.571411385,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:07.473951 4909 server/testserver.go:446  had 14 ranges at startup, expected 22
I180921 06:59:07.514999 5211 server/status/runtime.go:484  [n1] runtime stats: 684 MiB RSS, 184 goroutines, 15 MiB/18 MiB/46 MiB GO alloc/idle/total, 33 MiB/74 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (116x), 0.0/0.0/0 (%r/%w/q)disk, 194 KiB/194 KiB (r/w)net
I180921 06:59:07.878922 5150 storage/replica_proposal.go:211  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=3 start=1537513137.484636441,0 epo=1 pro=1537513147.876161804,0 following repl=(n1,s1):1 seq=2 start=1537513137.484636441,0 exp=1537513146.498892815,0 pro=1537513137.498921649,0
I180921 06:59:08.495209 4909 server/testserver.go:484  had 14 ranges at startup, expected 22
goroutine 4909 [running]:
github.com/cockroachdb/cockroach/pkg/server.WaitForInitialSplits(0xc421cd0580, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:478 +0x183
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).WaitForInitialSplits(0xc4207ee0e0, 0x3772780, 0xc4214b64e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:446 +0x44
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4207ee0e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:363 +0x30c
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:206 +0x141
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x379f540, 0xc4210422d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:174 +0x74
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.TestAdjustStartKeyForInterleave(0xc4210422d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/table_test.go:836 +0xc6
testing.tRunner(0xc4210422d0, 0x3104240)
    /usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:824 +0x565

goroutine 1 [chan receive]:
testing.(*T).Run(0xc42024d2c0, 0x30047f8, 0x1f, 0x3104240, 0xc420291c01)
    /usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc42024d2c0)
    /usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc42024d2c0, 0xc420291d78)
    /usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc4202d3300, 0x4712c00, 0x2c, 0x2c, 0xbee143838423f266)
    /usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc4202f4200, 0x0)
    /usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/sql/sqlbase_test.TestMain(0xc4202f4200)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/main_test.go:32 +0x100
main.main()
    _testmain.go:134 +0x22b

goroutine 20 [syscall]:
os/signal.signal_recv(0x89fef1)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0x132
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x1d4

goroutine 7 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x1ec

goroutine 21 [select, locked to thread]:
runtime.gopark(0x3106e38, 0x0, 0x2fd5f02, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc42008ff50, 0xc4200c62a0)
    /usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 5340 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc42177a0f0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x205
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x4d

goroutine 4913 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4218ca000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc4218ca000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 3699 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420021900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5141 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xae62a22b00000123)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac780)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac780)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde8a0, 0xc420f6c000, 0xc421fde890)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4990 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4203d6a80, 0x30, 0x2ea5360, 0x100000000000001, 0xc420a162d0)
    <autogenerated>:1 +0x80
crypto/tls.(*listener).Accept(0xc4203d6ac0, 0xc4200dc050, 0x2cf4ce0, 0x46f1750, 0x2f92220)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x53
net/http.(*Server).Serve(0xc420e17930, 0x375e3c0, 0xc4203d6ac0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x3772780, 0xc420a16270)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1170 +0x89
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde610, 0xc420f6c000, 0xc4203d6b00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4983 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x3772780, 0xc420a16240)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1627 +0x200
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde400, 0xc420f6c000, 0xc4212f30e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5115 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x9d7373ec0000010f)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac360)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac360)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde390, 0xc420f6c000, 0xc421fde380)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4964 [semacquire]:
sync.runtime_notifyListWait(0xc420a1b650, 0x0)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420a1b640)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420a1b520)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:655 +0x86
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:643 +0x607

goroutine 5153 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x721bd7c400000138)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc420ca8060)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc420ca8060)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdea90, 0xc420f6c000, 0xc421fdea80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 2225 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42044d180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5013 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420910200)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:869 +0x31d
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:281 +0x1651

goroutine 4988 [IO wait]:
internal/poll.runtime_pollWait(0x2b5dda2a1af0, 0x72, 0xc42008ecb8)
    /usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc420752398, 0x72, 0xc420b97700, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc420752398, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Accept(0xc420752380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:372 +0x2e2
net.(*netFD).accept(0xc420752380, 0xc4218fc010, 0x2b5dda2b25d0, 0xc420887b00)
    /usr/local/go/src/net/fd_unix.go:238 +0x53
net.(*TCPListener).accept(0xc420ed26a0, 0xc420754630, 0xc420e0fec8, 0x86b9dd)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x4e
net.(*TCPListener).Accept(0xc420ed26a0, 0x3104b20, 0xc420e2dbc0, 0xc4218fc010, 0xc4215c1c40)
    /usr/local/go/src/net/tcpsock.go:259 +0x50
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420e2dbc0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xc1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x3772780, 0xc420a16360)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1151 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde5f0, 0xc420f6c000, 0xc4203d6aa0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5215 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x3772780, 0xc4212f3f50)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:151 +0x1ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034aa30, 0xc420f6c000, 0xc42034aa20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 113 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42044d540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 114 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42044d680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5174 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3772780, 0xc420ca8600)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fded40, 0xc420f6c000, 0xc421cca360)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5214 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3772780, 0xc420a16840)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1326 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034a9f0, 0xc420f6c000, 0xc4210c5b00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4965 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3772780, 0xc420a16000)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde1a0, 0xc420f6c000, 0xc421fde190)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5147 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x721bd7c40000012e)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4202295c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4202295c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde990, 0xc420f6c000, 0xc421fde980)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5121 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xe83fd0dc00000112)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac480)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac480)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde520, 0xc420f6c000, 0xc421fde510)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4974 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3772780, 0xc420a160f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde2d0, 0xc420f6c000, 0xc421fde2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4992 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4203d6960, 0x31060a0, 0xc42100c1c0, 0x378b100, 0xc420ddec80)
    <autogenerated>:1 +0x80
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc42100c1c0, 0x37700c0, 0xc4203d6960, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:544 +0x2e1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x3772780, 0xc420a16570)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1186 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde630, 0xc420f6c000, 0xc4203d6b20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5179 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3772780, 0xc420ca8630)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fded90, 0xc420f6c000, 0xc421cca400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5188 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x3772780, 0xc420ca8d20)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x8a9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b75c0, 0xc420f6c000, 0xc4202b75b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5170 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3772780, 0xc4211acd50)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1676 +0x488
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdece0, 0xc420f6c000, 0xc420890370)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5131 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xdb2be13900000119)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac690)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac690)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde710, 0xc420f6c000, 0xc421fde6f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5216 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x3772780, 0xc420d4e960)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:299 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034aa50, 0xc420f6c000, 0xc4210c5b20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5144 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x8d0ccde30000012a)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac8d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac8d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde920, 0xc420f6c000, 0xc421fde8f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5138 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xbe286ce900000125)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac840)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac840)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde830, 0xc420f6c000, 0xc421fde820)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4987 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x3772780, 0xc420a16330)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1139 +0x84
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde5e0, 0xc420f6c000, 0xc4203d6a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5157 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x71208c590000013a)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc420ca80c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc420ca80c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdeb50, 0xc420f6c000, 0xc421fdeb40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4986 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3772780, 0xc420a16300)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:101 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde4c0, 0xc420f6c000, 0xc4203d6900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5167 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x3772780, 0xc420ca82a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1529 +0x149
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdecb0, 0xc420f6c000, 0xc421cca2e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5110 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xe83fd0dc00000110)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac270)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac270)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde2f0, 0xc420f6c000, 0xc421fde2e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5220 [select]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x3772780, 0xc4212f3fb0)
    /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:298 +0x146
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034abf0, 0xc420f6c000, 0xc4211ad920)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4929 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x721bd7c400000106)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac1b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac1b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde250, 0xc420f6c000, 0xc421fde240)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4910 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3772780, 0xc420a16510)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:101 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f8a030, 0xc420f6c000, 0xc420422060)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4999 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3772780, 0xc420228ed0)
    /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1362 +0x5a5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde940, 0xc420f6c000, 0xc421fde930)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5168 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3772780, 0xc420ca82d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1676 +0x488
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdecc0, 0xc420f6c000, 0xc4208902d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5148 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xfe894e4000000132)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac9c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac9c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde9f0, 0xc420f6c000, 0xc421fde9a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 4973 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3772780, 0xc420a160c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde2b0, 0xc420f6c000, 0xc421fde2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5204 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x3772780, 0xc420a16810)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:178 +0x1a3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034a3b0, 0xc420f6c000, 0xc4210c4540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5142 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x9d7373ec0000012d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac930)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac930)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde8c0, 0xc420f6c000, 0xc421fde8b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5154 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xfe894e4000000134)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211aca50)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211aca50)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdeaf0, 0xc420f6c000, 0xc421fdeaa0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5343 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420806000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420806000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 5092 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x3772780, 0xc4212f3d70)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:205 +0x1ce
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f8a3b0, 0xc420f6c000, 0xc420dae540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5210 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3772780, 0xc4203e9bc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1820 +0x1f9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034a990, 0xc420f6c000, 0xc4210c5a80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5014 [select]:
net.(*pipe).read(0xc420752400, 0xc420e70000, 0x400, 0x400, 0x16a5, 0x2000, 0xc4210e1680)
    /usr/local/go/src/net/pipe.go:164 +0x290
net.(*pipe).Read(0xc420752400, 0xc420e70000, 0x400, 0x400, 0xc42039e400, 0xc4210e16b0, 0xb70d8a)
    /usr/local/go/src/net/pipe.go:147 +0x61
crypto/tls.(*block).readFromUntil(0xc420a16600, 0x2b5dda2b8ac8, 0xc420752400, 0x5, 0xc420752400, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc420f0c000, 0x3106f17, 0xc420f0c120, 0xc420e7000d)
    /usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc420f0c000, 0xc4210e8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc4206ce180, 0xc42147c038, 0x9, 0x9, 0x1060100000020, 0x8, 0xc3c569601b64b9e2)
    /usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x3749560, 0xc4206ce180, 0xc42147c038, 0x9, 0x9, 0x9, 0x3756800, 0xc4214c8c60, 0x886780)
    /usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x3749560, 0xc4206ce180, 0xc42147c038, 0x9, 0x9, 0x8, 0x8, 0x8a0fd2)
    /usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42147c038, 0x9, 0x9, 0x3749560, 0xc4206ce180, 0x0, 0x0, 0xc4203f1720, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42147c000, 0xc4203f1720, 0xc4203f1720, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420910200, 0xc4214c8c30, 0x31060f8)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:427 +0xd6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc42100c1c0, 0x378f060, 0xc420910200)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:676 +0x16f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:631 +0x54
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc42039e7e0, 0xc42100c1c0, 0x2b5ddc1a53d8, 0xc420910200)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:639 +0x35
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0x7c7

goroutine 5143 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x721bd7c40000012c)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac960)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac960)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde8e0, 0xc420f6c000, 0xc421fde8d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5180 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3772780, 0xc4212f3e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdeda0, 0xc420f6c000, 0xc421cca420)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5117 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xe83fd0dc00000111)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac3f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac3f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde420, 0xc420f6c000, 0xc421fde400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5106 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x1fdf119f00000109)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac150)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac150)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde270, 0xc420f6c000, 0xc421fde260)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 829 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420020b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 830 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420020c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 4966 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3772780, 0xc420315c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde1c0, 0xc420f6c000, 0xc421fde1b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5120 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x1fdf119f0000011d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac510)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac510)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde4e0, 0xc420f6c000, 0xc421fde4d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5166 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc421480580, 0x3772780, 0xc4211accf0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4051 +0x1a6
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x3772780, 0xc4211accf0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3983 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdec70, 0xc420f6c000, 0xc421fdec60)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5132 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xdb2be13900000122)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac750)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac750)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde730, 0xc420f6c000, 0xc421fde720)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5152 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0x1fdf119f00000129)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211aca20)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211aca20)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdea70, 0xc420f6c000, 0xc421fdea60)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5137 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xc51a5200000124)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211ac7e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211ac7e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fde810, 0xc420f6c000, 0xc421fde800)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5156 [semacquire]:
sync.runtime_notifyListWait(0xc420ebe190, 0xbe286ce900000135)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420ebe180)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201d2ea0, 0x3772780, 0xc4211aca80)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3772780, 0xc4211aca80)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421fdeb30, 0xc420f6c000, 0xc421fdeb20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5011 [select]:
net.(*pipe).read(0xc420752480, 0xc42097c400, 0x400, 0x400, 0x16a5, 0x2000, 0xc4202736f0)
    /usr/local/go/src/net/pipe.go:164 +0x290
net.(*pipe).Read(0xc420752480, 0xc42097c400, 0x400, 0x400, 0xc4216879c0, 0xc420273720, 0xb70d8a)
    /usr/local/go/src/net/pipe.go:147 +0x61
crypto/tls.(*block).readFromUntil(0xc4214c8090, 0x2b5dda2b8ac8, 0xc420752480, 0x5, 0xc420752480, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc420f0c700, 0x3106f17, 0xc420f0c820, 0xc42097c40d)
    /usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc420f0c700, 0xc421fd6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc42184a840, 0xc4203fcc78, 0x9, 0x9, 0x2b5de2c49588, 0xc420e57bc0, 0x87d100)
    /usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x3749560, 0xc42184a840, 0xc4203fcc78, 0x9, 0x9, 0x9, 0xc420e57bc0, 0xc420e22180, 0xc420273cc8)
    /usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x3749560, 0xc42184a840, 0xc4203fcc78, 0x9, 0x9, 0x87d06a, 0xc42085fb00, 0x8a0fd2)
    /usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203fcc78, 0x9, 0x9, 0x3749560, 0xc42184a840, 0x0, 0x0, 0xc420e57bc0, 0xc420273da7)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203fcc40, 0xc4203f1440, 0xc4203f1400, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4209ba000)
tbg commented 6 years ago

@petermattis we see a few splits right away and then nothing for 10s (except a few lease acquisitions around the 8s mark, perhaps that means something).

petermattis commented 6 years ago

This again happened on release-2.1 where my recent addition of verbose split-queue logging in tests is not present. Closing this as a duplicate of #29144. 🤞 that we get a reproduction with that extra logging.