sql: TestTxnAutoRetry failed under stress

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/5cc40fe7c6d010b9ec1803396fb13e9916ef8899

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestTxnAutoRetry PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=902174&tab=buildLog

W180913 07:52:08.822897 143362 server/status/runtime.go:310  [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180913 07:52:08.876549 143362 server/server.go:851  [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I180913 07:52:08.877311 143362 base/addr_validation.go:279  [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180913 07:52:08.877439 143362 base/addr_validation.go:319  [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180913 07:52:08.883694 143362 server/config.go:493  [n?] 1 storage engine initialized
I180913 07:52:08.883946 143362 server/config.go:496  [n?] RocksDB cache size: 128 MiB
I180913 07:52:08.884080 143362 server/config.go:496  [n?] store 0: in-memory, size 0 B
I180913 07:52:09.012211 143362 server/node.go:374  [n?] **** cluster f877a5b7-54c2-407f-bc39-68dc1b419f64 has been created
I180913 07:52:09.012535 143362 server/server.go:1413  [n?] **** add additional nodes by specifying --join=127.0.0.1:46275
I180913 07:52:09.014361 143362 gossip/gossip.go:408  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:46275" > attrs:<> locality:<> ServerVersion:<major_val:2 minor_val:0 patch:0 unstable:12 > build_tag:"v2.1.0-beta.20180904-453-g5cc40fe" started_at:1536825129013773901 
I180913 07:52:09.020093 143246 gossip/gossip.go:942  [n1] gossip connectivity

I180913 07:52:09.050286 143362 storage/store.go:1558  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180913 07:52:09.054848 143362 server/node.go:477  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=6.9 KiB), ranges=1, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=7103.00 p25=7103.00 p50=7103.00 p75=7103.00 p90=7103.00 pMax=7103.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I180913 07:52:09.055585 143362 storage/stores.go:242  [n1] read 0 node addresses from persistent storage
I180913 07:52:09.056382 143362 server/node.go:700  [n1] connecting to gossip network to verify cluster ID...
I180913 07:52:09.056727 143362 server/node.go:725  [n1] node connected via gossip and verified as part of cluster "f877a5b7-54c2-407f-bc39-68dc1b419f64"
I180913 07:52:09.057186 143362 server/node.go:549  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180913 07:52:09.068681 143362 server/status/recorder.go:652  [n1] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180913 07:52:09.068934 143362 server/server.go:1804  [n1] Could not start heap profiler worker due to: directory to store profiles could not be determined
I180913 07:52:09.070942 143362 server/server.go:1550  [n1] starting https server at 127.0.0.1:33659 (use: 127.0.0.1:33659)
I180913 07:52:09.071142 143362 server/server.go:1552  [n1] starting grpc/postgres server at 127.0.0.1:46275
I180913 07:52:09.071228 143362 server/server.go:1553  [n1] advertising CockroachDB node at 127.0.0.1:46275
I180913 07:52:09.089597 143587 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180913 07:52:09.140640 143293 storage/replica_command.go:298  [n1,split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180913 07:52:09.375375 143361 storage/replica_command.go:298  [n1,split,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180913 07:52:09.541196 143312 sql/event_log.go:126  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I180913 07:52:09.568173 143596 storage/replica_command.go:298  [n1,split,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
W180913 07:52:09.695818 143620 storage/intent_resolver.go:668  [n1,s1] failed to push during intent resolution: failed to push "split" id=a3adbd7b key=/Local/Range/System/NodeLiveness/RangeDescriptor rw=true pri=0.00222195 iso=SERIALIZABLE stat=PENDING epo=0 ts=1536825129.568474530,0 orig=1536825129.568474530,0 max=1536825129.568474530,0 wto=false rop=false seq=1
I180913 07:52:09.741504 143650 storage/replica_command.go:298  [n1,split,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180913 07:52:09.881573 143653 storage/replica_command.go:298  [n1,split,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180913 07:52:10.011910 143621 sql/event_log.go:126  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:2.0-12 User:root}
I180913 07:52:10.028452 143667 storage/replica_command.go:298  [n1,split,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180913 07:52:10.212816 143683 storage/replica_command.go:298  [n1,split,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180913 07:52:10.222908 143605 rpc/nodedialer/nodedialer.go:92  [n1,consistencyChecker,s1,r1/1:/{Min-System/}] connection to n1 established
I180913 07:52:10.359654 143626 sql/event_log.go:126  [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root}
I180913 07:52:10.390006 143675 storage/replica_command.go:298  [n1,split,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180913 07:52:10.477527 143733 storage/replica_command.go:298  [n1,split,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180913 07:52:10.568165 143722 storage/replica_command.go:298  [n1,split,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180913 07:52:10.667091 143703 storage/replica_command.go:298  [n1,split,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180913 07:52:10.682832 143690 sql/event_log.go:126  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:1d9d31be-34c7-4736-9829-8fa8d6e4a1a5 User:root}
I180913 07:52:10.750921 143735 storage/replica_command.go:298  [n1,split,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180913 07:52:10.887467 143246 gossip/gossip.go:942  [n1] gossip connectivity
  n1 [sentinel];

I180913 07:52:10.907764 143665 storage/replica_command.go:298  [n1,split,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180913 07:52:10.915232 143580 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I180913 07:52:11.022243 143810 storage/replica_command.go:298  [n1,split,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180913 07:52:11.105511 143726 sql/event_log.go:126  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I180913 07:52:11.164897 143362 server/server.go:1606  [n1] done ensuring all necessary migrations have run
I180913 07:52:11.165224 143362 server/server.go:1609  [n1] serving sql connections
I180913 07:52:11.202484 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.205525 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.207173 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.215219 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.225946 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.228162 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.229494 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.230923 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.232271 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.233696 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.235145 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.236856 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.238015 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.240431 143712 sql/event_log.go:126  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:46275} Attrs: Locality: ServerVersion:2.0-12 BuildTag:v2.1.0-beta.20180904-453-g5cc40fe StartedAt:1536825129013773901 LocalityAddress:[]} ClusterID:f877a5b7-54c2-407f-bc39-68dc1b419f64 StartedAt:1536825129013773901 LastUp:1536825129013773901}
I180913 07:52:11.242166 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.243789 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.244698 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.251033 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.253776 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.254550 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.255378 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.261681 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.264149 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.267122 143710 server/server_update.go:67  [n1] no need to upgrade, cluster already at the newest version
I180913 07:52:11.268310 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.273802 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.282956 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.304378 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.339682 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.408253 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.544014 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:11.814190 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:12.352924 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:13.354847 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:14.357830 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:15.359912 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:16.362025 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:17.364071 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:18.094680 143562 storage/replica_proposal.go:211  [n1,s1,r8/1:/Table/1{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1536825129.021207693,0 epo=1 pro=1536825138.089414922,0 following repl=(n1,s1):1 seq=2 start=1536825129.021207693,0 exp=1536825138.038992729,0 pro=1536825129.039091300,0
I180913 07:52:18.366222 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:19.075822 143505 server/status/runtime.go:484  [n1] runtime stats: 1.3 GiB RSS, 610 goroutines, 21 MiB/75 MiB/120 MiB GO alloc/idle/total, 69 MiB/136 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (1687x), 0.0/0.0/0 (%r/%w/q)disk, 526 KiB/526 KiB (r/w)net
I180913 07:52:19.097289 143563 storage/replica_proposal.go:211  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=3 start=1536825129.021207693,0 epo=1 pro=1536825139.091968433,0 following repl=(n1,s1):1 seq=2 start=1536825129.021207693,0 exp=1536825138.038992729,0 pro=1536825129.039091300,0
I180913 07:52:19.211192 143587 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180913 07:52:19.344634 143510 storage/replica_proposal.go:211  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=3 start=1536825129.021207693,0 epo=1 pro=1536825139.340610025,0 following repl=(n1,s1):1 seq=2 start=1536825129.021207693,0 exp=1536825138.038992729,0 pro=1536825129.039091300,0
I180913 07:52:19.368317 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:19.510576 143508 storage/replica_proposal.go:211  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=3 start=1536825129.021207693,0 epo=1 pro=1536825139.505183145,0 following repl=(n1,s1):1 seq=2 start=1536825129.021207693,0 exp=1536825138.038992729,0 pro=1536825129.039091300,0
I180913 07:52:20.370139 143362 server/testserver.go:420  had 15 ranges at startup, expected 22
I180913 07:52:21.384667 143362 server/testserver.go:458  had 15 ranges at startup, expected 22
goroutine 143362 [running]:
github.com/cockroachdb/cockroach/pkg/server.WaitForInitialSplits(0xc420806e80, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:452 +0x183
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).WaitForInitialSplits(0xc425801810, 0x3a46b00, 0xc424118cf0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:420 +0x44
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc425801810, 0x3a1c140, 0xc420eab450, 0x0, 0x0, 0x3a1a0a0, 0xc4215d48c0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:363 +0x30c
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x3a1c140, 0xc420eab450, 0x0, 0x0, 0x3a1a0a0, 0xc4215d48c0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:206 +0x141
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x3a75bc0, 0xc424200d20, 0x3a1c140, 0xc420eab450, 0x0, 0x0, 0x3a1a0a0, 0xc4215d48c0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:174 +0x74
github.com/cockroachdb/cockroach/pkg/sql_test.TestTxnAutoRetry(0xc424200d20)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/txn_restart_test.go:455 +0x24d
testing.tRunner(0xc424200d20, 0x33727b0)
    /usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:824 +0x565

goroutine 1 [chan receive]:
testing.(*T).Run(0xc42020b1d0, 0x3242737, 0x10, 0x33727b0, 0xc420625c01)
    /usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc42020b1d0)
    /usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc42020b1d0, 0xc420625d68)
    /usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc4202fbe80, 0x4a85160, 0xe9, 0xe9, 0xbedea2a68bfdebc4)
    /usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc4204b6a00, 0x0)
    /usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc4204b6a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0x15e
main.main()
    _testmain.go:514 +0x22b

goroutine 6 [syscall, 17 minutes]:
os/signal.signal_recv(0x89fef1)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 50 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0x132
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x1d4

goroutine 51 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x1ec

goroutine 8 [select, 17 minutes, locked to thread]:
runtime.gopark(0x3375320, 0x0, 0x323112d, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc42008ff50, 0xc42050c120)
    /usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 143133 [select]:
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc422538e00, 0x3a46b00, 0xc4213ff650)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:130 +0x411
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).(github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.runCloser)-fm(0x3a46b00, 0xc4213ff650)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:88 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420a04440, 0xc422753170, 0xc420a04420)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 8841 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420de9a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 132419 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a8d680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 41107 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421b77180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 3106 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421575900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 118518 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a8ca00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 20817 [chan receive, 15 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c0c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 66417 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262d400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 117410 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a5d680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 68656 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421575e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 43019 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42037f7c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 86 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c1540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 87 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c1680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 97593 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4229fd900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 69845 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262d180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 60511 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420ad0640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 68102 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420de8780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 27871 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42312c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 78582 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e9680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143531 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x9a04158000000121)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc421683c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc421683c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c63e0, 0xc422753170, 0xc4202c63d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143588 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3a46b00, 0xc422541560)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1326 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204a34b0, 0xc422753170, 0xc4221b9c00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 74584 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262d900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 43705 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e9400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143562 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x9a0415800000013e)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420626480)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420626480)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6a90, 0xc422753170, 0xc4202c6a80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143393 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a46b00, 0xc421947d10)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:90 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203feb80, 0xc422753170, 0xc4203feb70)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 137181 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424f34b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 27201 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42346eb40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 83966 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262c500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 48379 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420ad1a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 30818 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e8c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 130085 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42548e8c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 29362 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4223ff540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143526 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x9a04158000000124)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420626120)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420626120)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c62d0, 0xc422753170, 0xc4202c62b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 132418 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a8d540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 39079 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42346ec80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143609 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42274e000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc42274e000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 16678 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e8780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 130084 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42548e780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 84546 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4231588c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 23471 [chan receive, 15 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42312cdc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 9496 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262c640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 72442 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c0f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 39619 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c17c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 46977 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421b76000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 9186 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205c1e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 98221 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42076bcc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 104293 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a5c500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 7911 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42346f2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 33237 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42312c780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 7137 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421574c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 81596 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a8cdc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 32626 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4223fe8c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 52184 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420ad08c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 39881 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4229fc640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143405 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x3a46b00, 0xc422762f30)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1173 +0xbe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ff020, 0xc422753170, 0xc420811780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143561 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x1a42861800000136)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc4206264e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc4206264e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6a70, 0xc422753170, 0xc4202c6a60)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 124312 [chan receive, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424f343c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 69269 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4223fe000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 33806 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421574280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 110027 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424f35e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143529 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x5f10560d00000122)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420626240)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420626240)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6390, 0xc422753170, 0xc4202c6370)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 92618 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421e43680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 2576 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42076b400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 2577 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42076b540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 44265 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420146c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 75309 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42312d900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143537 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x1f23839d00000127)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420b71560)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420b71560)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6550, 0xc422753170, 0xc4202c6540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 3699 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421575b80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 92617 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421e43540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 85115 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4218b0f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 15434 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42346fa40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 142390 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213f0640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 114654 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a8c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 14699 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42262c000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143541 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x1f23839d00000129)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420b717d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420b717d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c65d0, 0xc422753170, 0xc4202c65c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143404 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4203d50a0, 0x30, 0x30f8e20, 0x100000000000001, 0xc421947e60)
    <autogenerated>:1 +0x80
crypto/tls.(*listener).Accept(0xc4203d50e0, 0xc420052148, 0x2f31320, 0x4a5b760, 0x31ebe80)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x53
net/http.(*Server).Serve(0xc420f53a00, 0x3a2f8c0, 0xc4203d50e0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x3a46b00, 0xc421947e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1169 +0x89
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ff010, 0xc422753170, 0xc4203d5100)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143516 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x1a4286180000014d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc421683b30)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc421683b30)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6140, 0xc422753170, 0xc4202c6130)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143558 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0xa36dd9890000013c)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420626660)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420626660)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6a10, 0xc422753170, 0xc4202c6a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 77044 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4229fd2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 24705 [chan receive, 15 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420ad1400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143264 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0xfdedfb1600000148)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc4213ff740)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc4213ff740)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420349e00, 0xc422753170, 0xc420349dd0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143630 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc420cade40, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc420bc0b40, 0xc425279270, 0xc420cade40)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc4210da000)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0x1ab
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0x162c

goroutine 39880 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4229fc500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 12570 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420de9900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 85973 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4229fcdc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143502 [select]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc4231d4690, 0xc425248700, 0xc4204a32b0, 0x33756b0, 0xc422c1baa0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:954 +0x20f
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x3a46b00, 0xc4236118f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1028 +0xdb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204a32c0, 0xc422753170, 0xc4234d4450)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 86499 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a5dcc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 100406 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421574000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 104292 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421a5c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 41108 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421b772c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143261 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x5f10560d0000013f)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc4213ff680)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc4213ff680)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420349a90, 0xc422753170, 0xc420349a80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 86754 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42114c3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 25310 [chan receive, 15 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42312db80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 79162 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421b76c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143530 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0x6c0e848e00000123)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc420626270)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc420626270)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c63b0, 0xc422753170, 0xc4202c63a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 143263 [semacquire]:
sync.runtime_notifyListWait(0xc4210b1710, 0xb95edcf500000139)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4210b1700)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4231eddd0, 0x3a46b00, 0xc4214659b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a46b00, 0xc4214659b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420349bb0, 0xc422753170, 0xc420349b10)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 133488 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4256e72c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 124920 [chan receive, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421b77900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 57248 [chan receive, 11 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420ad1e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 41911 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42346e280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 713 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42076a640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 714 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42076a780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:81 +0x47c

goroutine 143245 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3a46b00, 0xc422763110)
    /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1364 +0x5a5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4220053b0, 0xc422753170, 0xc4220053a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 113299 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423159400)

[jordanlewis]

Dupe of #30181.