search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

tests: had 15 ranges at startup, expected 22 #30181

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/4d3d9f7e2eec5579c49e25025d2443698347ec20

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestPGUnwrapError PKG=github.com/cockroachdb/cockroach/pkg/sql/pgwire TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=902269&tab=buildLog


W180913 08:45:58.486617 5336 server/status/runtime.go:310  [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180913 08:45:58.526876 5336 server/server.go:851  [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I180913 08:45:58.527401 5336 base/addr_validation.go:279  [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180913 08:45:58.527491 5336 base/addr_validation.go:319  [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180913 08:45:58.532577 5336 server/config.go:493  [n?] 1 storage engine initialized
I180913 08:45:58.532675 5336 server/config.go:496  [n?] RocksDB cache size: 128 MiB
I180913 08:45:58.532749 5336 server/config.go:496  [n?] store 0: in-memory, size 0 B
I180913 08:45:58.581054 5336 server/node.go:374  [n?] **** cluster aa4bfacf-6286-4f37-be1c-5b2bcbb2cda0 has been created
I180913 08:45:58.581258 5336 server/server.go:1413  [n?] **** add additional nodes by specifying --join=127.0.0.1:36887
I180913 08:45:58.582307 5336 gossip/gossip.go:408  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:36887" > attrs:<> locality:<> ServerVersion:<major_val:2 minor_val:0 patch:0 unstable:13 > build_tag:"v2.2.0-alpha.00000000-780-g4d3d9f7" started_at:1536828358581884564 
I180913 08:45:58.585649 5474 gossip/gossip.go:942  [n1] gossip connectivity

I180913 08:45:58.605206 5336 server/node.go:477  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=6.9 KiB), ranges=1, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=7103.00 p25=7103.00 p50=7103.00 p75=7103.00 p90=7103.00 pMax=7103.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I180913 08:45:58.606948 5336 storage/stores.go:242  [n1] read 0 node addresses from persistent storage
I180913 08:45:58.607516 5336 server/node.go:700  [n1] connecting to gossip network to verify cluster ID...
I180913 08:45:58.607668 5336 server/node.go:725  [n1] node connected via gossip and verified as part of cluster "aa4bfacf-6286-4f37-be1c-5b2bcbb2cda0"
I180913 08:45:58.610297 5336 server/node.go:549  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180913 08:45:58.611351 5336 server/status/recorder.go:611  [n1] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180913 08:45:58.611428 5336 server/server.go:1804  [n1] Could not start heap profiler worker due to: directory to store profiles could not be determined
I180913 08:45:58.613211 5336 server/server.go:1550  [n1] starting https server at 127.0.0.1:42235 (use: 127.0.0.1:42235)
I180913 08:45:58.613298 5336 server/server.go:1552  [n1] starting grpc/postgres server at 127.0.0.1:36887
I180913 08:45:58.613356 5336 server/server.go:1553  [n1] advertising CockroachDB node at 127.0.0.1:36887
I180913 08:45:58.620456 5341 storage/replica_command.go:298  [n1,split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180913 08:45:58.666724 5382 storage/replica_command.go:298  [n1,split,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180913 08:45:58.802999 5508 storage/replica_command.go:298  [n1,split,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
W180913 08:45:58.814898 5343 storage/intent_resolver.go:668  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=229f5b2c key=/Table/SystemConfigSpan/Start rw=true pri=0.00364920 iso=SERIALIZABLE stat=PENDING epo=0 ts=1536828358.639855205,0 orig=1536828358.639855205,0 max=1536828358.639855205,0 wto=false rop=false seq=6
I180913 08:45:58.835480 5665 sql/event_log.go:126  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I180913 08:45:58.857366 5511 storage/replica_command.go:298  [n1,split,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180913 08:45:58.905642 5464 storage/replica_command.go:298  [n1,split,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180913 08:45:58.963128 5383 sql/event_log.go:126  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:2.0-13 User:root}
I180913 08:45:58.976833 5686 storage/replica_command.go:298  [n1,split,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180913 08:45:59.027019 5677 storage/replica_command.go:298  [n1,split,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180913 08:45:59.053409 5685 sql/event_log.go:126  [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root}
I180913 08:45:59.070579 5467 storage/replica_command.go:298  [n1,split,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180913 08:45:59.100126 5409 storage/replica_command.go:298  [n1,split,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180913 08:45:59.128548 5388 storage/replica_command.go:298  [n1,split,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180913 08:45:59.155872 5709 storage/replica_command.go:298  [n1,split,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180913 08:45:59.184167 5732 sql/event_log.go:126  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:9e38105c-8094-4cf6-93c1-804ccaf4d64b User:root}
I180913 08:45:59.185420 5713 storage/replica_command.go:298  [n1,split,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180913 08:45:59.215757 5727 storage/replica_command.go:298  [n1,split,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180913 08:45:59.230026 5754 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I180913 08:45:59.244567 5782 storage/replica_command.go:298  [n1,split,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180913 08:45:59.257599 5781 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I180913 08:45:59.278512 5336 server/server.go:1606  [n1] done ensuring all necessary migrations have run
I180913 08:45:59.278634 5336 server/server.go:1609  [n1] serving sql connections
I180913 08:45:59.283477 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.284268 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.284912 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.285567 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.286253 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.286890 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.287458 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.288073 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.288633 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.289258 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.289912 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.290587 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.295182 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.295695 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.296090 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.296191 5765 server/server_update.go:67  [n1] no need to upgrade, cluster already at the newest version
I180913 08:45:59.296623 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.297351 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.297498 5767 sql/event_log.go:126  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:36887} Attrs: Locality: ServerVersion:2.0-13 BuildTag:v2.2.0-alpha.00000000-780-g4d3d9f7 StartedAt:1536828358581884564 LocalityAddress:[]} ClusterID:aa4bfacf-6286-4f37-be1c-5b2bcbb2cda0 StartedAt:1536828358581884564 LastUp:1536828358581884564}
I180913 08:45:59.298150 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.298835 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.299555 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.300530 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.302051 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.304637 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.309430 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.318654 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.336428 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.370918 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.439063 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.574360 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:45:59.689449 5789 rpc/nodedialer/nodedialer.go:92  [n1,consistencyChecker,s1,r1/1:/{Min-System/}] connection to n1 established
I180913 08:45:59.843816 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:00.381803 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:00.543554 5474 gossip/gossip.go:942  [n1] gossip connectivity
  n1 [sentinel];

I180913 08:46:01.384010 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:02.385022 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:03.386127 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:04.387282 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:05.388437 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:06.389479 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:07.390685 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:07.620374 5894 util/protoutil/randnullability.go:94  inserting null for (roachpb.Lease).Expiration: false
I180913 08:46:07.620516 5894 util/protoutil/randnullability.go:94  inserting null for (roachpb.Lease).DeprecatedStartStasis: true
I180913 08:46:07.623394 5591 storage/replica_proposal.go:211  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=3 start=1536828358.591500369,0 epo=1 pro=1536828367.619517710,0 following repl=(n1,s1):1 seq=2 start=1536828358.591500369,0 exp=1536828367.603706904,0 pro=1536828358.603746377,0
I180913 08:46:08.391837 5336 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 08:46:08.614608 5652 server/status/runtime.go:484  [n1] runtime stats: 409 MiB RSS, 189 goroutines, 11 MiB/18 MiB/42 MiB GO alloc/idle/total, 34 MiB/73 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (88x), 0.0/0.0/0 (%r/%w/q)disk, 891 KiB/891 KiB (r/w)net
I180913 08:46:08.623824 5597 storage/replica_proposal.go:211  [n1,s1,r8/1:/Table/1{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1536828358.591500369,0 epo=1 pro=1536828368.620862044,0 following repl=(n1,s1):1 seq=2 start=1536828358.591500369,0 exp=1536828367.603706904,0 pro=1536828358.603746377,0
I180913 08:46:08.957141 5602 storage/replica_proposal.go:211  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=3 start=1536828358.591500369,0 epo=1 pro=1536828368.954649934,0 following repl=(n1,s1):1 seq=2 start=1536828358.591500369,0 exp=1536828367.603706904,0 pro=1536828358.603746377,0
I180913 08:46:09.025127 5605 storage/replica_proposal.go:211  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=3 start=1536828358.591500369,0 epo=1 pro=1536828369.022111429,0 following repl=(n1,s1):1 seq=2 start=1536828358.591500369,0 exp=1536828367.603706904,0 pro=1536828358.603746377,0
I180913 08:46:09.402509 5336 server/testserver.go:458  had 15 ranges at startup, expected 22
goroutine 5336 [running]:
github.com/cockroachdb/cockroach/pkg/server.WaitForInitialSplits(0xc4217f5500, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:452 +0x183
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).WaitForInitialSplits(0xc4213632d0, 0x38f3f20, 0xc420ccc720)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:420 +0x44
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4213632d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:363 +0x30c
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:206 +0x141
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x39206a0, 0xc4210ec3c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:174 +0x74
github.com/cockroachdb/cockroach/pkg/sql/pgwire_test.TestPGUnwrapError(0xc4210ec3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwire_test.go:423 +0xc6
testing.tRunner(0xc4210ec3c0, 0x320d4e0)
    /usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:824 +0x565

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201f94a0, 0x30f0b77, 0x11, 0x320d4e0, 0xc4205cfc01)
    /usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc4201f94a0)
    /usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc4201f94a0, 0xc4205cfd80)
    /usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc420314080, 0x48d88e0, 0x2b, 0x2b, 0xbedea6cd487a7e07)
    /usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc420213580, 0x0)
    /usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/sql/pgwire_test.TestMain(0xc420213580)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/main_test.go:37 +0xd6
main.main()
    _testmain.go:170 +0x22b

goroutine 36 [syscall]:
os/signal.signal_recv(0x89fef1)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0x132
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x1d4

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x1ec

goroutine 43 [select, locked to thread]:
runtime.gopark(0x3210fb0, 0x0, 0x30de5f0, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc42008ff50, 0xc4200c82a0)
    /usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 5624 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x38f3f20, 0xc420bcf2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1529 +0x149
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f240, 0xc4218fb050, 0xc420350820)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 796 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ad180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5603 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x413144670000012e)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c390)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c390)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038eeb0, 0xc4218fb050, 0xc42038eea0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 232 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201ef900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 233 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201efa40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5585 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x413144670000011c)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc4214d23f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc4214d23f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ebf0, 0xc4218fb050, 0xc42038ebe0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5361 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x38f3f20, 0xc421106000)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f280, 0xc4218fb050, 0xc420d5f270)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5582 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x44ab27280000011b)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc4214d2330)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc4214d2330)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038eb70, 0xc4218fb050, 0xc42038eb50)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5592 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x42fd3f7e00000127)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c150)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c150)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ed50, 0xc4218fb050, 0xc42038ed40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5445 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x38f3f20, 0xc421010270)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1138 +0x84
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f650, 0xc4218fb050, 0xc420314d20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 797 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ad2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5655 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x38f3f20, 0xc4211063c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1326 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c28e0, 0xc4218fb050, 0xc420423760)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5441 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x38f3f20, 0xc421106150)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1627 +0x200
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f480, 0xc4218fb050, 0xc420bcfad0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5617 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x2e2b17f4000000f7)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c630)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c630)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f140, 0xc4218fb050, 0xc42038f130)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5412 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x38f3f20, 0xc420bce150)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:155 +0x63
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e7e0, 0xc4218fb050, 0xc4203507e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5660 [select]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x38f3f20, 0xc4214d3770)
    /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:281 +0x16a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c2c00, 0xc4218fb050, 0xc4214d31a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5424 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xe2a6193c00000108)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc420bceed0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc420bceed0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e960, 0xc4218fb050, 0xc42038e950)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5847 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420f74200)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:869 +0x31d
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:281 +0x1651

goroutine 5619 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xd552970300000101)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c6f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c6f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f180, 0xc4218fb050, 0xc42038f170)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5874 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc4208ba100, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc420bfa3c0, 0xc42049e240, 0xc4208ba100)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc4200dafc0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x101
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0x14cf

goroutine 1479 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ac640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5400 [select]:
github.com/cockroachdb/cockroach/pkg/kv.GRPCTransportFactory.func1(0x38f3ea0, 0xc4200e4058)
    /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport_race.go:120 +0x427
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4218fb050, 0x38f3ea0, 0xc4200e4058, 0xc421392200, 0x17, 0x0, 0x0, 0xc420378230)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:319 +0x14d

goroutine 5652 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func2(0x38f3f20, 0xc42076cbd0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1835 +0x1a3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c27e0, 0xc4218fb050, 0xc420423700)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5631 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x38f3f20, 0xc420bcf320)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1748 +0x39a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f2c0, 0xc4218fb050, 0xc420350860)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5650 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x38f3f20, 0xc42076cba0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:112 +0x350
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c24c0, 0xc4218fb050, 0xc4204223e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5605 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xd55297030000012f)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c3f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c3f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038eef0, 0xc4218fb050, 0xc42038eee0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5639 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38f3f20, 0xc42076ca20)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f3e0, 0xc4218fb050, 0xc420350b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5790 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc420d76930)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x205
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x4d

goroutine 5606 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xd552970300000130)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c420)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c420)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ef10, 0xc4218fb050, 0xc42038ef00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5644 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38f3f20, 0xc420bcf470)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f430, 0xc4218fb050, 0xc420350be0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5587 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x413144670000011d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc4214d2450)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc4214d2450)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ecb0, 0xc4218fb050, 0xc42038eca0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5574 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x9242417400000110)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc420bcf080)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc420bcf080)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ea20, 0xc4218fb050, 0xc42038ea10)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5640 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38f3f20, 0xc42076ca50)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f3f0, 0xc4218fb050, 0xc420350b60)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5594 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x42fd3f7e00000124)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c1b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c1b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ed90, 0xc4218fb050, 0xc42038ed80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5455 [semacquire]:
sync.runtime_notifyListWait(0xc4211b3e70, 0xcb)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4211b3e60)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4211b3d40)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:656 +0x86
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:644 +0x607

goroutine 5447 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420314d40, 0xc4200e4050, 0x2defcc0, 0x48b7770, 0x3099980)
    <autogenerated>:1 +0x80
net/http.(*Server).Serve(0xc420a12000, 0x38efe60, 0xc420314d40, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x38f3f20, 0xc4210102a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1162 +0x146
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f670, 0xc4218fb050, 0xc420bcfd70)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5450 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420314c40, 0x3210218, 0xc4209df880, 0x390ca20, 0xc420fb6140)
    <autogenerated>:1 +0x80
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4209df880, 0x38efe60, 0xc420314c40, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:544 +0x2e1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x38f3f20, 0xc421106210)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1185 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f6a0, 0xc4218fb050, 0xc420314de0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5595 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x9242417400000125)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c1e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c1e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038edb0, 0xc4218fb050, 0xc42038eda0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5454 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x38f3f20, 0xc421010570)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1311 +0x84
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f850, 0xc4218fb050, 0xc420315120)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5599 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x4131446700000126)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ee30, 0xc4218fb050, 0xc42038ee20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5651 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x38f3f20, 0xc421521530)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1819 +0x1f9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c2750, 0xc4218fb050, 0xc4204236e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5427 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x38f3f20, 0xc421106030)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f2c0, 0xc4218fb050, 0xc420d5f2b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5610 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x4b831ee9000000f6)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c4e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c4e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038efc0, 0xc4218fb050, 0xc42038efa0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5636 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38f3f20, 0xc420bcf3b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f3b0, 0xc4218fb050, 0xc420350ac0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5659 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x38f3f20, 0xc4211063f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:502 +0x48e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c2bf0, 0xc4218fb050, 0xc4211185f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5430 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x38f3f20, 0xc421010090)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f320, 0xc4218fb050, 0xc420d5f310)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 1480 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ac780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5446 [IO wait]:
internal/poll.runtime_pollWait(0x2aed1b826c90, 0x72, 0xc42008ccb8)
    /usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc42096cf18, 0x72, 0xc420e2d900, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc42096cf18, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Accept(0xc42096cf00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:372 +0x2e2
net.(*netFD).accept(0xc42096cf00, 0xc420424010, 0x2aed1b834fa0, 0xc4200b9200)
    /usr/local/go/src/net/fd_unix.go:238 +0x53
net.(*TCPListener).accept(0xc42068e328, 0xc421142cb0, 0xc420ac26c8, 0x86b9dd)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x4e
net.(*TCPListener).Accept(0xc42068e328, 0x320ebf0, 0xc420ba4b80, 0xc420424010, 0xc42117cc00)
    /usr/local/go/src/net/tcpsock.go:259 +0x50
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420ba4b80, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xc1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x38f3f20, 0xc4211061e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1150 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f660, 0xc4218fb050, 0xc420314d80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5875 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc4200dafc0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1176 +0x1ad
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:309 +0x1558

goroutine 2025 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ad400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5375 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x38f3f20, 0xc421106360)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:791 +0x174
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c23f0, 0xc4218fb050, 0xc420422340)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5423 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xe2a6193c00000107)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc420bceea0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc420bceea0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e940, 0xc4218fb050, 0xc42038e930)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5611 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x92424174000000f5)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c510)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c510)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038eff0, 0xc4218fb050, 0xc42038efd0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 2026 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4217ad540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5440 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420773400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5421 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x9242417400000104)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc420bcee40)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc420bcee40)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e900, 0xc4218fb050, 0xc42038e8f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5604 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xd552970300000132)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038eed0, 0xc4218fb050, 0xc42038eec0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5425 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xe9fb96390000010d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc420bcef90)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc420bcef90)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e980, 0xc4218fb050, 0xc42038e970)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5609 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0xe2a6193c000000f4)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c4b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c4b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038ef90, 0xc4218fb050, 0xc42038ef70)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5618 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x41314467000000fc)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203babd0, 0x38f3f20, 0xc42185c660)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38f3f20, 0xc42185c660)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f160, 0xc4218fb050, 0xc42038f150)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5623 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc420154b00, 0x38f3f20, 0xc420bcf1a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4089 +0x1a6
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x38f3f20, 0xc420bcf1a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4021 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038f200, 0xc4218fb050, 0xc42038f1f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5474 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x38f3f20, 0xc4210105a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1399 +0x41d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f9d0, 0xc4218fb050, 0xc420d5f9c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5438 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x38f3f20, 0xc421010210)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f420, 0xc4218fb050, 0xc420d5f410)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5380 [select]:
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc421534080, 0x38f3f20, 0xc42185c690)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:130 +0x411
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).(github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.runCloser)-fm(0x38f3f20, 0xc42185c690)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:88 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038e460, 0xc4218fb050, 0xc42038e450)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5451 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func9(0x38f3f20, 0xc421010510)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1281 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420d5f780, 0xc4218fb050, 0xc4217e80c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 3179 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420c30780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 5634 [select]:
net.(*pipe).read(0xc42096d000, 0xc42086a400, 0x400, 0x400, 0x16a5, 0x2000, 0xc4205cd6f0)
    /usr/local/go/src/net/pipe.go:164 +0x290
net.(*pipe).Read(0xc42096d000, 0xc42086a400, 0x400, 0x400, 0xc420380180, 0xc4205cd720, 0xc06e6a)
    /usr/local/go/src/net/pipe.go:147 +0x61
crypto/tls.(*block).readFromUntil(0xc42185c0f0, 0x2aed1d764a60, 0xc42096d000, 0x5, 0xc42096d000, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc420569500, 0x3211117, 0xc420569620, 0xc42086a40d)
    /usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc420569500, 0xc4208d0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc4208cf7a0, 0xc42035a038, 0x9, 0x9, 0x2aed33bdb248, 0xc420543aa0, 0x87d100)
    /usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x38c7a60, 0xc4208cf7a0, 0xc42035a038, 0x9, 0x9, 0x9, 0xc420543aa0, 0xc421574000, 0xc4205cdcc8)
    /usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x38c7a60, 0xc4208cf7a0, 0xc42035a038, 0x9, 0x9, 0x87d06a, 0xc4206cc000, 0x8a0fd2)
    /usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42035a038, 0x9, 0x9, 0x38c7a60, 0xc4208cf7a0, 0x0, 0x0, 0xc420543aa0, 0xc4205cdda7)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42035a000, 0xc4203d2000, 0xc4203d2000, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4200dab40)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x1fe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x10fc

goroutine 5656 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x38f3f20, 0xc420bcf4d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:151 +0x1ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c2900, 0xc4218fb050, 0xc4202c28f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 5583 [semacquire]:
sync.runtime_notifyListWait(0xc421564110, 0x44ab272800000119)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421564100)
    /usr/local/go/src/sync/cond.go:56 +0x8e
jordanlewis commented 6 years ago 
I180913 08:46:09.402509 5336 server/testserver.go:458  had 15 ranges at startup, expected 22

I thought this got solved? cc @petermattis and renaming to capture other similar issues

petermattis commented 6 years ago

I have been unable to reproduce any of these failures locally. I did add more logging which is showing that the cluster is not getting wedged (log lines are still being output). Perhaps the split hung.

Duplicate of #29144.