search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

sql: TestTxnUserRestart failed under stress #29421

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/b33f4fb8b22b7bc11935fda980edbdaabde5f3ef

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestTxnUserRestart PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=874858&tab=buildLog


=== RUN   TestTxnUserRestart/err=RETRY_POSSIBLE_REPLAY,stgy=1
W180831 07:49:35.093269 149958 server/status/runtime.go:310  [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180831 07:49:35.308475 149958 server/server.go:849  [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I180831 07:49:35.309532 149958 base/addr_validation.go:279  [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180831 07:49:35.309674 149958 base/addr_validation.go:319  [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I180831 07:49:35.316185 149958 server/config.go:493  [n?] 1 storage engine initialized
I180831 07:49:35.316300 149958 server/config.go:496  [n?] RocksDB cache size: 128 MiB
I180831 07:49:35.316393 149958 server/config.go:496  [n?] store 0: in-memory, size 0 B
I180831 07:49:35.431588 149958 server/node.go:373  [n?] **** cluster 1155630d-c0e7-4884-9a9a-cee689fc1cad has been created
I180831 07:49:35.431850 149958 server/server.go:1411  [n?] **** add additional nodes by specifying --join=127.0.0.1:34813
I180831 07:49:35.433255 149958 gossip/gossip.go:407  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:34813" > attrs:<> locality:<> ServerVersion:<major_val:2 minor_val:0 patch:0 unstable:12 > build_tag:"v2.1.0-alpha.20180702-2222-gb33f4fb" started_at:1535701775432747157 
I180831 07:49:35.440510 150014 gossip/gossip.go:941  [n1] gossip connectivity

I180831 07:49:35.544961 149958 storage/store.go:1540  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180831 07:49:35.550209 149958 server/node.go:476  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=6.9 KiB), ranges=1, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=7103.00 p25=7103.00 p50=7103.00 p75=7103.00 p90=7103.00 pMax=7103.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I180831 07:49:35.551012 149958 storage/stores.go:242  [n1] read 0 node addresses from persistent storage
I180831 07:49:35.551968 149958 server/node.go:697  [n1] connecting to gossip network to verify cluster ID...
I180831 07:49:35.552336 149958 server/node.go:722  [n1] node connected via gossip and verified as part of cluster "1155630d-c0e7-4884-9a9a-cee689fc1cad"
I180831 07:49:35.567598 149958 server/node.go:546  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180831 07:49:35.569129 149958 server/status/recorder.go:611  [n1] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180831 07:49:35.569239 149958 server/server.go:1802  [n1] Could not start heap profiler worker due to: directory to store profiles could not be determined
I180831 07:49:35.601342 149958 server/server.go:1548  [n1] starting https server at 127.0.0.1:34255 (use: 127.0.0.1:34255)
I180831 07:49:35.601586 149958 server/server.go:1550  [n1] starting grpc/postgres server at 127.0.0.1:34813
I180831 07:49:35.601642 149958 server/server.go:1551  [n1] advertising CockroachDB node at 127.0.0.1:34813
W180831 07:49:35.601988 149958 jobs/registry.go:309  [n1] unable to get node liveness: node not in the liveness table
I180831 07:49:35.656583 149860 storage/replica_command.go:298  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
panic: test timed out after 40m0s

goroutine 149944 [running]:
testing.(*M).startAlarm.func1()
    /usr/local/go/src/testing/testing.go:1240 +0x146
created by time.goFunc
    /usr/local/go/src/time/sleep.go:172 +0x52

goroutine 1 [chan receive]:
testing.(*T).Run(0xc42022b1d0, 0x321cb51, 0x12, 0x3348818, 0xc42063bc01)
    /usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc42022b1d0)
    /usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc42022b1d0, 0xc42063bd68)
    /usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc420349020, 0x4a51040, 0xe7, 0xe7, 0xbeda584c16f5a224)
    /usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc420398a00, 0x0)
    /usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc420398a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0x15e
main.main()
    _testmain.go:510 +0x22b

goroutine 20 [syscall, 40 minutes]:
os/signal.signal_recv(0x89fef1)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 4 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0x132
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x1d4

goroutine 5 [chan receive, 40 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x1ec

goroutine 29 [select, 40 minutes, locked to thread]:
runtime.gopark(0x334b368, 0x0, 0x320793f, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc42008ff50, 0xc42021e060)
    /usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 4222 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201763c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 60208 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cb2280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 2623 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f1680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 105258 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42171b900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 134739 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422092280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 134119 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f32c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 2622 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f1540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 9061 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f1040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 77209 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acc500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 71 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053b540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 72 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053b680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 99963 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4255ff040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 47187 [chan receive, 30 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423718b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 79333 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423718280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 13324 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42367e000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149342 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4277fa000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 7509 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4214bd400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 77587 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b6a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 72078 [chan receive, 24 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216be8c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 6182 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216bf680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 14791 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203a1e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 102599 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42171a000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 106380 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423bbc280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 5597 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f03c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 83862 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b63c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 3706 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053bb80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 42647 [chan receive, 32 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203a1cc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 70057 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235f7b80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 49529 [chan receive, 30 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053af00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 69446 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a803c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 35001 [chan receive, 33 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a80f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150159 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x3a17160, 0xc420dfa9f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:750 +0x357
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013bf0, 0xc424ca5dd0, 0xc42385d240)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 149965 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x3a17160, 0xc4212bc1e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1627 +0x200
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040c0b0, 0xc424ca5dd0, 0xc4216dea50)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 26233 [chan receive, 35 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f0dc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 6945 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216be280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150125 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x2b)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc42176a2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc42176a2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013590, 0xc424ca5dd0, 0xc423013580)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 43778 [chan receive, 31 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423275900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 4223 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420176640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 81861 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acc000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 9887 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4214bc3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 17101 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42367edc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 53778 [chan receive, 28 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6f180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149994 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a17160, 0xc4216de180)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c910, 0xc424ca5dd0, 0xc42031c900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 78191 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422427540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 90902 [chan receive, 20 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4277fb540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 70058 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235f7cc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 104648 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b7b80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 82410 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a81a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150101 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x12)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc421791cb0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc421791cb0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013290, 0xc424ca5dd0, 0xc423013280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 32783 [chan receive, 34 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ed400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 84451 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b7540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 81310 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6e140)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 78467 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cb2c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149990 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a17160, 0xc4216de120)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c760, 0xc424ca5dd0, 0xc42031c740)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 77210 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acc640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 77884 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6f680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 37701 [chan receive, 33 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216beb40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150091 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0xc7a1e34c00000048)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc421791b30)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc421791b30)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013150, 0xc424ca5dd0, 0xc423013140)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 65305 [chan receive, 26 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423719680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 145729 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f2000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150003 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4228b8240, 0xc4200dc050, 0x2f0a200, 0x4a27760, 0x31c1e00)
    <autogenerated>:1 +0x80
net/http.(*Server).Serve(0xc420e40c30, 0x3a12fe0, 0xc4228b8240, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x3a17160, 0xc424446030)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1160 +0x146
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040c3a0, 0xc424ca5dd0, 0xc4216def90)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 66964 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423275540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 128752 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4277fa8c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 60772 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42367e780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 83306 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203a17c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 3134 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f0500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 21769 [chan receive, 36 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a80500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 114206 [chan receive, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b6640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150139 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x38)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc42081c4b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc42081c4b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013750, 0xc424ca5dd0, 0xc423013740)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 150087 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x43)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc421791ad0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc421791ad0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4230130d0, 0xc424ca5dd0, 0xc4230130c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 150094 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0xb)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc420daa7b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc420daa7b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4230131b0, 0xc424ca5dd0, 0xc4230131a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 58497 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6f400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 115986 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42171b040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 81309 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6e000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 105548 [chan receive, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42171bcc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 47802 [chan receive, 30 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ec780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 66417 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acdcc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150152 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3a17160, 0xc423d92030)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1650 +0x488
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013900, 0xc424ca5dd0, 0xc420b59860)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 90903 [chan receive, 20 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4277fb680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149970 [IO wait]:
internal/poll.runtime_pollWait(0x2b3d9f17a3c0, 0x72, 0xc42277fcb8)
    /usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc420bf2618, 0x72, 0xc4226fe300, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc420bf2618, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Accept(0xc420bf2600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:372 +0x2e2
net.(*netFD).accept(0xc420bf2600, 0xc420e2a010, 0x2b3da03527d8, 0xc42161b500)
    /usr/local/go/src/net/fd_unix.go:238 +0x53
net.(*TCPListener).accept(0xc42000e078, 0xc420f02530, 0xc427d9aec8, 0x86b9dd)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x4e
net.(*TCPListener).Accept(0xc42000e078, 0x3348fb8, 0xc420f9e240, 0xc420e2a010, 0xc424731780)
    /usr/local/go/src/net/tcpsock.go:259 +0x50
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420f9e240, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xc1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x3a17160, 0xc4212bc3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1148 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040c390, 0xc424ca5dd0, 0xc4228b82e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 150112 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x1d)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc420daa960)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc420daa960)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4230133f0, 0xc424ca5dd0, 0xc4230133e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 138662 [chan receive, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423bbc640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 46023 [chan receive, 31 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4214bca00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 42022 [chan receive, 32 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cb2140)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 690 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f08c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 691 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f0c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 10993 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4232748c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 145426 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422427040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150109 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x1a)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc421791d70)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc421791d70)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013390, 0xc424ca5dd0, 0xc423013380)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 149989 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a17160, 0xc4212bc120)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c6d0, 0xc424ca5dd0, 0xc42031c6c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 150088 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x45)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc420daa6c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc420daa6c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4230130f0, 0xc424ca5dd0, 0xc4230130e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 72859 [chan receive, 24 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ec280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 111656 [chan receive, 15 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ec500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 23802 [chan receive, 35 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42367f7c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 46601 [chan receive, 31 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acd040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 16003 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4214bc280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 35561 [chan receive, 33 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423274b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150182 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3a17160, 0xc423d92db0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1325 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013e00, 0xc424ca5dd0, 0xc4203d06a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 84141 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42171b680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149959 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc424ca5dd0, 0xc421872630)
    /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:180 +0x11d
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
    /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:177 +0x57

goroutine 82411 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a81b80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 61051 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424b6e780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 63712 [chan receive, 26 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421accc80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 9060 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4206f0f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150133 [semacquire]:
sync.runtime_notifyListWait(0xc420c34c10, 0x2f)
    /usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c34c00)
    /usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4213dd290, 0x3a17160, 0xc42081c2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a17160, 0xc42081c2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423013690, 0xc424ca5dd0, 0xc423013680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 59061 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a81680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 83863 [chan receive, 22 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4213b6500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 9372 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053b900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150012 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x3a17160, 0xc42081c0f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:369 +0x83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040c7d0, 0xc424ca5dd0, 0xc4228b8f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 62058 [chan receive, 26 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423719900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 143800 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423cf6780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 74559 [chan receive, 24 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ecdc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 24362 [chan receive, 35 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053b040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 75769 [chan receive, 23 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423719540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 98816 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422427cc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 73450 [chan receive, 24 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235f7680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 23801 [chan receive, 35 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42367f680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 63161 [chan receive, 26 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acd680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 9886 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4214bc140)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 71191 [chan receive, 24 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acd2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 69445 [chan receive, 25 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a80000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 59062 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a817c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 22657 [chan receive, 36 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216bf2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 150004 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4228b82a0, 0x30, 0x30d0660, 0x100000000000001, 0xc4212bc300)
    <autogenerated>:1 +0x80
crypto/tls.(*listener).Accept(0xc4228b8300, 0xc4200dc050, 0x2f0a200, 0x4a27760, 0x31c1e00)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x53
net/http.(*Server).Serve(0xc422c0e000, 0x39fffe0, 0xc4228b8300, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x3a17160, 0xc4212bc2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1167 +0x89
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040c3b0, 0xc424ca5dd0, 0xc4228b8320)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 14067 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4216bec80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149246 [chan receive]:
testing.(*T).Run(0xc4235f4b40, 0xc420400440, 0x20, 0xc424c59460, 0x2)
    /usr/local/go/src/testing/testing.go:825 +0x597
github.com/cockroachdb/cockroach/pkg/sql_test.TestTxnUserRestart(0xc4235f4b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/txn_restart_test.go:949 +0x4bb
testing.tRunner(0xc4235f4b40, 0x3348818)
    /usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:824 +0x565

goroutine 91992 [chan receive, 20 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421acc780)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 56989 [chan receive, 27 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42053a500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x47c

goroutine 149958 [chan receive]:
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).execInternal(0xc420fd8ff0, 0x3a17160, 0xc424446930, 0x323c34c, 0x1f, 0x0, 0xc426178c00, 0x3288bbb, 0x38, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:497 +0x991
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Exec(0xc420fd8ff0, 0x3a17160, 0xc422b73680, 0x323c34c, 0x1f, 0x0, 0x3288bbb, 0x38, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:292 +0x1d2
github.com/cockroachdb/cockroach/pkg/sqlmigrations.runStmtAsRootWithRetry(0x3a17160, 0xc422b73680, 0x3a17360, 0xc423d83080, 0xc420fd8ff0, 0x323c34c, 0x1f, 0x3288bbb, 0x38, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:516 +0x27d
github.com/cockroachdb/cockroach/pkg/sqlmigrations.optInToDiagnosticsStatReporting(0x3a17160, 0xc422b73680, 0x3a17360, 0xc423d83080, 0xc420fd8ff0, 0x0, 0x1)
    /go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:539 +0xfb
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations(0xc42100a140, 0x3a17160, 0xc422b73680, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:449 +0xe32
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc421075800, 0x3a17160, 0xc422b73680, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1595 +0x4e7e
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4225845b0, 0x39eca60, 0xc4217c9d40, 0x0, 0x0, 0x39ea9c0, 0xc42377b640, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:350 +0x2c0
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x39eca60, 0xc4217c9d40, 0x0, 0x0, 0x39ea9c0, 0xc42377b640, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:206 +0x141
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x3a464a0, 0xc4221de690, 0x39eca60, 0xc4217c9d40, 0x0, 0x0, 0x39ea9c0, 0xc42377b640, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:174 +0x74
github.com/cockroachdb/cockroach/pkg/sql_test.TestTxnUserRestart.func1(0xc4221de690)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/txn_restart_test.go:955 +0x23a
testing.tRunner(0xc4221de690, 0xc424c59460)
    /usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:824 +0x565

goroutine 149993 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a17160, 0xc4212bc180)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c8f0, 0xc424ca5dd0, 0xc42031c870)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb

goroutine 64799 [chan receive, 26 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208ecc80)
cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/14b53bcf4501e9117155eed1653c113a74e267cb

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestTxnUserRestart PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=879358&tab=buildLog

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/19942d06f60abbc20175fb456f8e718b05a75b7f

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestTxnUserRestart PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=894518&tab=buildLog

andreimatei commented 6 years ago

these are all timeouts from before we reduced the stressrace parallelism. Closing.