search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.04k stars 3.8k forks source link

sql: TestSplitAt failed under stress #29169

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/dbf5d6ed987c2c240bcbf0bb2b514c8e2186f855

Parameters:

TAGS=
GOFLAGS=

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestSplitAt PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=865443&tab=buildLog


I180828 05:45:51.285589 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:01.086616 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 20 MiB/7.1 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 19.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:46:01.285617 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:11.087152 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 14.90cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:46:11.285638 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:21.089295 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 20 MiB/7.0 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 16.70cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:46:21.285598 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:31.086650 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 21.91cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:46:31.285588 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:41.090927 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 21 MiB/6.9 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 18.19cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:46:41.285580 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:46:51.085964 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:46:51.086668 165531 server/status/runtime.go:433  [n1] runtime stats: 216 MiB RSS, 586 goroutines, 14 MiB/13 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 17.01cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:46:51.285620 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:01.086678 165531 server/status/runtime.go:433  [n1] runtime stats: 218 MiB RSS, 586 goroutines, 20 MiB/7.9 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 20.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:47:01.285614 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:11.086712 165531 server/status/runtime.go:433  [n1] runtime stats: 218 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 18.40cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:47:11.285578 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:21.086675 165531 server/status/runtime.go:433  [n1] runtime stats: 218 MiB RSS, 586 goroutines, 21 MiB/6.7 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 28.40cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:47:21.285642 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:31.086673 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 19.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:47:31.285646 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:41.086656 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 21 MiB/6.8 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 16.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:47:41.285640 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:47:51.085967 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:47:51.086698 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 14 MiB/13 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 19.50cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:47:51.286457 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:01.086857 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 20 MiB/7.5 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 20.40cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:48:01.285662 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:11.087283 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 14.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:48:11.285596 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:21.086696 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 21 MiB/6.9 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 16.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:48:21.285561 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:31.086699 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 18.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:48:31.285667 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:41.086649 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 20 MiB/7.3 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 19.40cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:48:41.285607 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:48:51.085962 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:48:51.086664 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 14 MiB/13 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 19.50cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:48:51.285637 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:01.086739 165531 server/status/runtime.go:433  [n1] runtime stats: 220 MiB RSS, 586 goroutines, 20 MiB/7.9 MiB/51 MiB GO alloc/idle/total, 78 MiB/108 MiB CGO alloc/total, 20.30cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:49:01.285607 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:11.086598 165531 server/status/runtime.go:433  [n1] runtime stats: 222 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 14.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:49:11.285590 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:21.087061 165531 server/status/runtime.go:433  [n1] runtime stats: 222 MiB RSS, 586 goroutines, 20 MiB/7.6 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 18.30cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:49:21.285591 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:31.086650 165531 server/status/runtime.go:433  [n1] runtime stats: 222 MiB RSS, 586 goroutines, 15 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 21.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:49:31.285657 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:41.086688 165531 server/status/runtime.go:433  [n1] runtime stats: 223 MiB RSS, 586 goroutines, 21 MiB/6.7 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 16.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:49:41.285593 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:49:51.085947 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:49:51.086603 165531 server/status/runtime.go:433  [n1] runtime stats: 225 MiB RSS, 586 goroutines, 14 MiB/13 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 32.10cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:49:51.285644 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:01.086646 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 20 MiB/7.6 MiB/50 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 19.30cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:50:01.285612 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:11.086724 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 15 MiB/11 MiB/50 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 18.40cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:50:11.285654 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:21.086758 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 21 MiB/6.0 MiB/50 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 16.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:50:21.285657 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:31.086827 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 14 MiB/13 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 18.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:50:31.285599 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:41.086587 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 19 MiB/7.8 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 15.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:50:41.285635 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:50:51.085971 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:50:51.086707 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 19.50cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:50:51.285597 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:01.086625 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 20 MiB/6.9 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 23.90cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:51:01.285627 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:11.086725 165531 server/status/runtime.go:433  [n1] runtime stats: 224 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 14.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:51:11.285621 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:21.086706 165531 server/status/runtime.go:433  [n1] runtime stats: 226 MiB RSS, 586 goroutines, 20 MiB/7.0 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 15.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:51:21.285620 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:31.086711 165531 server/status/runtime.go:433  [n1] runtime stats: 226 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 20.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:51:31.285662 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:41.086624 165531 server/status/runtime.go:433  [n1] runtime stats: 226 MiB RSS, 586 goroutines, 20 MiB/6.7 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 20.40cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:51:41.285592 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:51:51.086034 165528 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180828 05:51:51.090176 165531 server/status/runtime.go:433  [n1] runtime stats: 226 MiB RSS, 586 goroutines, 14 MiB/12 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 19.49cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:51:51.285698 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:52:01.087025 165531 server/status/runtime.go:433  [n1] runtime stats: 226 MiB RSS, 586 goroutines, 20 MiB/7.0 MiB/51 MiB GO alloc/idle/total, 86 MiB/116 MiB CGO alloc/total, 19.31cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:52:01.285610 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:52:11.086786 165531 server/status/runtime.go:433  [n1] runtime stats: 227 MiB RSS, 586 goroutines, 15 MiB/11 MiB/51 MiB GO alloc/idle/total, 126 MiB/155 MiB CGO alloc/total, 15.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180828 05:52:11.285601 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180828 05:52:21.086668 165531 server/status/runtime.go:433  [n1] runtime stats: 227 MiB RSS, 586 goroutines, 21 MiB/6.5 MiB/51 MiB GO alloc/idle/total, 126 MiB/155 MiB CGO alloc/total, 29.40cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180828 05:52:21.285573 165533 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
panic: test timed out after 40m0s

goroutine 168079 [running]:
testing.(*M).startAlarm.func1()
    /usr/local/go/src/testing/testing.go:1240 +0xfc
created by time.goFunc
    /usr/local/go/src/time/sleep.go:172 +0x44

goroutine 1 [chan receive, 37 minutes]:
testing.(*T).Run(0xc4225573b0, 0x25997c9, 0xb, 0x26d0f08, 0x8b9201)
    /usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc42022d2c0)
    /usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc42022d2c0, 0xc420761dd8)
    /usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc4204d4200, 0x39a59c0, 0xe6, 0xe6, 0xbed9544d70a8388e)
    /usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc4203a2e00, 0x0)
    /usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc4203a2e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0xb8
main.main()
    _testmain.go:508 +0x151

goroutine 20 [syscall, 40 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 25 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x126

goroutine 26 [chan receive, 40 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x13e

goroutine 34 [select, 40 minutes, locked to thread]:
runtime.gopark(0x26d39c8, 0x0, 0x2590d5a, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc4204e0750, 0xc4204fe060)
    /usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 48557 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4219bb680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 25412 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f2280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 131086 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f43680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 73500 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4219bb040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 77160 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423633e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165432 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xc11)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc422b1af00)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc422b1af00)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ae940, 0xc4220395f0, 0xc4203ae920)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 75322 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42283f2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 28870 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f3040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165442 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xc1e)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc4211724e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc4211724e0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203aead0, 0xc4220395f0, 0xc4203aeab0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 75 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205cda40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 76 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205cdb80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165952 [select, 37 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc4224e39c0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x14a
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x14f

goroutine 47296 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423632dc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 3788 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421865a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165444 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbf9)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc421a88000)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc421a88000)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203aeb30, 0xc4220395f0, 0xc4203aeb10)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 112984 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156ef00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 32915 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201757c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 26732 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421139400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 9095 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421865400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165417 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbff)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc421172060)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc421172060)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ae670, 0xc4220395f0, 0xc4203ae650)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 165466 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbec)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc422b1b200)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc422b1b200)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203af090, 0xc4220395f0, 0xc4203af080)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 53153 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f27400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 144686 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42283e500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 112467 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421864b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 74370 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421865040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 52329 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423284a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 7530 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f2a00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 123892 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423870b40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 53154 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f27540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 80202 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156fa40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165169 [IO wait, 37 minutes]:
internal/poll.runtime_pollWait(0x2ab192d8de38, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4203a2398, 0x72, 0xc422b30b00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4203a2398, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc4203a2380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc4203a2380, 0xc421215708, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc422c38270, 0x882980, 0xc4214b7e88, 0xc4214b7e90)
    /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc422c38270, 0x26d1770, 0xc420a3df80, 0x2ae3600, 0xc421215708)
    /usr/local/go/src/net/tcpsock.go:259 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420a3df80, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func14.1()
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1420 +0x2f
sync.(*Once).Do(0xc4203f8580, 0xc421b88760)
    /usr/local/go/src/sync/once.go:44 +0xbe
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func14(0x2acc620, 0xc421172030)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1419 +0x5b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ae080, 0xc4220395f0, 0xc4217ce000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 84153 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4219bb180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 112983 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156edc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 70536 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424945180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 161871 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cecf00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 86267 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4218657c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 94862 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f13900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165293 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2acc620, 0xc424441f80)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239cef80, 0xc4220395f0, 0xc4239cef70)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 165462 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbf2)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc421172630)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc421172630)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203aef30, 0xc4220395f0, 0xc4203aef20)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 160650 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422500f00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 3215 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203b1e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 159723 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156fcc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 10060 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421139180)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 94252 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156fb80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 33856 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4219bb900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 5959 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421864c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 121971 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4238712c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 161235 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205cd400)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 49195 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e2500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165540 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2acc620, 0xc422094510)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:502 +0x3fc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421ec32d0, 0xc4220395f0, 0xc423c6a550)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 3787 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421865900)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165476 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbfc)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc421a88030)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc421a88030)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203af1e0, 0xc4220395f0, 0xc4203af1d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 83526 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42283f040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 81455 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4220e3b80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 139859 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cecb40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165527 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x2acc620, 0xc421eb8120)
    /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:788 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204524c0, 0xc4220395f0, 0xc420abc2a0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 56975 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424945680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 64905 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421864280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 66996 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423285540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165499 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2acc620, 0xc422b1b5c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200d1ec0, 0xc4220395f0, 0xc4238384c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 92090 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42156ec80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 92122 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f26000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 23818 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421139680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 39317 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4236337c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165418 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xc02)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc4211721b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc4211721b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ae690, 0xc4220395f0, 0xc4203ae680)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 131085 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f43540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 9776 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4218643c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165472 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbfe)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc4211726f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc4211726f0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203af160, 0xc4220395f0, 0xc4203af150)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 165295 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2acc620, 0xc42343e000)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239cefc0, 0xc4220395f0, 0xc4239cefb0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 134320 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422cec000)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165953 [select, 37 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc421683b80)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x23b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc421683b80)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x1b5
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0xe1

goroutine 165296 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2acc620, 0xc42343e0c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239cefe0, 0xc4220395f0, 0xc4239cefd0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 165480 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x2acc620, 0xc421a88060)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1510 +0x10e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203af280, 0xc4220395f0, 0xc4217ce980)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 165290 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2acc620, 0xc424441ec0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239cef20, 0xc4220395f0, 0xc4239cef10)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 9096 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421865540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 749 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f23c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 750 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f2500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 149607 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f423c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 22167 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421139a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165323 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc421565280, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x133
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc4211c5ce0, 0xc4203de430, 0xc421565280)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x166
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc424f0f200)
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x80
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0xc8f

goroutine 7824 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203b1a40)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165301 [select, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x26d3d30, 0xc42327b590, 0xc420210150, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:185 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
    /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:178 +0xaf

goroutine 88225 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4205cd2c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 120356 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423871e00)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165461 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xbf1)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc421172600)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc421172600)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203aef00, 0xc4220395f0, 0xc4203aeef0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 154775 [chan receive, 37 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4232843c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 10059 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421139040)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 56246 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4203f2dc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165453 [semacquire, 1 minutes]:
sync.runtime_notifyListWait(0xc424646110, 0xbe2)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc422b1b0b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc422b1b0b0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203aed00, 0xc4220395f0, 0xc4203aece0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 82086 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42283e280)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 1412 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421864640)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 1411 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421864500)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 67663 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421138c80)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 52912 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42075e3c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 61212 [chan receive, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4211383c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 124749 [chan receive, 38 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc424945540)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 165424 [semacquire]:
sync.runtime_notifyListWait(0xc424646110, 0xc0a)
    /usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc424646100)
    /usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420860d80, 0x2acc620, 0xc4211722d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2acc620, 0xc4211722d0)
    /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ae7f0, 0xc4220395f0, 0xc4203ae790)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
maddyblue commented 6 years ago

I'm just gonna assume this should be assigned to you.

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/dbf5d6ed987c2c240bcbf0bb2b514c8e2186f855

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=TestSplitAt PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=865673&tab=buildLog

petermattis commented 6 years ago

I'm seeing this frequently in CI runs right now.

petermattis commented 6 years ago

Reproduces with: make stress PKG=./sql TESTS=TestSplitAt TESTTIMEOUT=1m. Not sure if that timeout is fair, though the test runs in ~0.3s without stress. Instrumentation shows that it is hanging on the very first test case: ALTER TABLE d.t SPLIT AT VALUES (2, 'b'). Logs show:

I180830 14:23:57.227447 39 sql/split_test.go:132  0: running: ALTER TABLE d.t SPLIT AT VALUES (2, 'b')
I180830 14:23:57.228020 675 storage/replica_command.go:298  [n1,s1,r27/1:/Table/5{3/1/1/"a"-4}] initiating a split of this range at key /Table/53/1/2/"b" [r28]
...

And then not much else. Goroutine 675 has a beast of a stack:

goroutine 675 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4207ce580, 0x65a44e0, 0xc4213e55f0, 0x154faf9175331855, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3179 +0xd93
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc4206984d0, 0x65a44e0, 0xc4213e5590, 0x0, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:185 +0xdb
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x65a44e0, 0xc4213e5590, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1006 +0x19a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc42058ed80, 0x65a44e0, 0xc4213e5590, 0x607b306, 0x10, 0xc4211438f0, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:303 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc420511180, 0x65a44e0, 0xc4213e5590, 0xc4213ee700, 0xc4213e5590, 0x0, 0xc421143b78)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:993 +0x201
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc420511180, 0x65a44e0, 0xc4213e5590, 0xc4213ee700, 0x0, 0xc421143bf0, 0x4126f4c)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1034 +0x9c
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(0x65842a0, 0xc420511180, 0x65a44e0, 0xc4213e5560, 0xc4213ee700, 0x0, 0x0, 0x0, 0xc420f72ed0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:375 +0x4f
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc4213e5530, 0x65a44e0, 0xc4213e5560, 0x6588960, 0xc420267c30, 0x0, 0x0, 0x100000001, 0x1, 0x1b, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x138
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc4213e5530, 0x65a44e0, 0xc420f72ed0, 0x0, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:169 +0x138
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0xc420540950, 0x1b, 0xc4203df4a0, 0x1, 0x1, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1303 +0x30a
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0x1b, 0xc4203df4a0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:387 +0x252
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:454 +0x227
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1080 +0x322
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421243400, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:903 +0x88b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc420540900, 0x65a44e0, 0xc420f72ed0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:668 +0x4c7
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc4204bdfc0, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:232 +0xf5
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetrics).SendLocked(0xc4204bdf88, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metrics.go:57 +0x115
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc4204bdef0, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:153 +0x98
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc4204bdef0, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:105 +0x11e
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc4204bde70, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:161 +0x165
github.com/cockroachdb/cockroach/pkg/kv.(*txnIntentCollector).SendLocked(0xc4204bde30, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_intent_collector.go:106 +0x45d
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc4204bdf70, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_sequence_nums.go:68 +0x246
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeat).SendLocked(0xc4204bdd98, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeat.go:219 +0x53f
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc4204bdc00, 0x65a44e0, 0xc420f72e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421453600, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:600 +0x537
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc420706400, 0x65a44e0, 0xc420ef2720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:613 +0x135
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc42141d720, 0x65a44e0, 0xc420ef2720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:797 +0x161
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send-fm(0x65a44e0, 0xc420ef2720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:504 +0x83
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x65a44e0, 0xc420ef2720, 0xc4211468b8, 0xc4215dca00, 0x40, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:538 +0x108
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc42141d720, 0x65a44e0, 0xc420ef2720, 0xc4215dca00, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:504 +0xc7
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor.func1(0x65a44e0, 0xc420ef2720, 0xc42141d720, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:369 +0x5bd
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x65a44e0, 0xc420ef2720, 0xc42141d720, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:576 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc42141d720, 0x65a44e0, 0xc420ef2720, 0xc420467820, 0xc420ef2720, 0x1)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:692 +0xe6
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc420706400, 0x65a44e0, 0xc420ef2720, 0xc420ef2780, 0xc421146e80, 0x2)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:575 +0xd1
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc42134a700, 0x65a44e0, 0xc420ef2720, 0xc420276ed0, 0x7, 0x8, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:301 +0x92a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).AdminSplit(0xc42134a700, 0x65a44e0, 0xc420ef2720, 0xc420276ed0, 0x7, 0x8, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:171 +0x280
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc42134a700, 0x65a44e0, 0xc420ef2720, 0x154faf91753287f3, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2668 +0xba4
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc42134a700, 0x65a44e0, 0xc420ef2720, 0x1b, 0x154faf91753287f3, 0x0, 0x100000001, 0x1, 0x1b, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2028 +0x653
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc42134a700, 0x65a44e0, 0xc420ef26f0, 0x154faf91753287f3, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1970 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4207ce580, 0x65a44e0, 0xc420ef26f0, 0x154faf91753287f3, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3089 +0x606
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc4206984d0, 0x65a44e0, 0xc420ef2690, 0x0, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:185 +0xdb
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x65a44e0, 0xc420ef2690, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1006 +0x19a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc42058ed80, 0x65a44e0, 0xc420ef2690, 0x607b306, 0x10, 0xc421148028, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:303 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc420511180, 0x65a44e0, 0xc420ef2690, 0xc420e93d00, 0xc420ef2690, 0x402d0e9, 0x61ac130)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:993 +0x201
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc420511180, 0x65a44e0, 0xc420ef2690, 0xc420e93d00, 0x0, 0xc421148328, 0x4126f4c)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1034 +0x9c
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(0x65842a0, 0xc420511180, 0x65a44e0, 0xc420ef2660, 0xc420e93d00, 0x0, 0x0, 0x0, 0xc420ef25a0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:375 +0x4f
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc420ef2630, 0x65a44e0, 0xc420ef2660, 0x6588960, 0xc420267c30, 0x0, 0x0, 0x100000001, 0x1, 0x1b, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x138
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc420ef2630, 0x65a44e0, 0xc420ef25a0, 0x0, 0x0, 0x100000001, 0x1, 0x1b, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:169 +0x138
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0xc420540950, 0x1b, 0xc4203aa940, 0x1, 0x1, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1303 +0x30a
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0x1b, 0xc4203aa940, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:387 +0x252
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:454 +0x227
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1080 +0x322
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:756 +0x1364
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc420540900, 0x65a44e0, 0xc420ef25a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:668 +0x4c7
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc420706460, 0x65a44e0, 0xc420ef2480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:213 +0xc1
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc420706400, 0x65a44e0, 0xc420ef2480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:613 +0x135
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc420706400, 0x65a44e0, 0xc420ef2480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:595 +0xa4
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x65a44e0, 0xc420ef2480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:561 +0x83
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x65a44e0, 0xc420ef2480, 0xc421149bd0, 0xc42163ee00, 0xc420276ed0, 0xc420276ed0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:538 +0x108
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc420706400, 0x65a44e0, 0xc420ef2480, 0xc42163ee00, 0xc4203aa8c0, 0x1)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:561 +0x94
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminSplit(0xc420706400, 0x65a44e0, 0xc420ef2480, 0x5cc79c0, 0xc4203aa8a0, 0x5cc79c0, 0xc4203aa8c0, 0x8, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:476 +0x98
github.com/cockroachdb/cockroach/pkg/sql.(*splitNode).Next(0xc420f3bbc0, 0x65a44e0, 0xc420ef2480, 0xc42142bdc0, 0xc421598a38, 0x0, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/split.go:130 +0x229
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc42167a000, 0xc421598a38, 0x6598160, 0xc420f3bbc0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:189 +0x5d3
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x65a44e0, 0xc420ef2480, 0x65ad160, 0xc42167a000, 0x6584860, 0xc42034d880)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:170 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc42167a000, 0x65a44e0, 0xc420ef2480, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:731 +0x98
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).StartSync(0xc4213121c0, 0x65a44e0, 0xc420ef2480, 0x61a6940, 0xc420ee6540, 0x6584420)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:594 +0x191
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc420c93600, 0xc420f3bc20, 0xc42141c8c0, 0xc42114a808, 0xc421492500, 0xc421598ad0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:245 +0x879
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc420c93600, 0x65a44e0, 0xc420f10f90, 0xc421598ad0, 0xc420f3bc20, 0xc42141c8c0, 0x6598160, 0xc420f3bbc0, 0xc421492500)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:737 +0x24c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc421598600, 0x65a44e0, 0xc420f10f90, 0xc421598a38, 0x3, 0x9babad8, 0xc4213db050, 0x0, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:966 +0x2d8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc421598600, 0x65a44e0, 0xc420f10f90, 0x65a7fe0, 0xc4203aa040, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:808 +0x688
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc421598600, 0x65a44e0, 0xc420f10f90, 0x65a7fe0, 0xc4203aa040, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:401 +0xb82
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc421598600, 0x65a44e0, 0xc420f10f90, 0x65a7fe0, 0xc4203aa040, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:95 +0x358
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc421598600, 0x65a4420, 0xc420860bc0, 0xc42030e4e0, 0x0, 0x0)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1079 +0x2186
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc42051c240, 0x65a4420, 0xc420860bc0, 0x0, 0x0, 0xc42146c04a, 0x4, 0x0, 0x0, 0x6587720, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:416 +0x1bb
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func3(0xc42051c240, 0x65a4420, 0xc420860bc0, 0xc421095340, 0x5400, 0x15000, 0xc420130840, 0xc42030e4e0, 0xc42030e4d0, 0xc420377a10, ...)
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:267 +0x122
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl
    /Users/pmattis/Development/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:266 +0xf04
jordanlewis commented 6 years ago

Heh, I wonder if things like splitNode should force an async path in client.DB to avoid these giant stacks... it's not clear to me whether it's okay that the SQL goroutines end up being involved in stuff like this.

petermattis commented 6 years ago

Goroutine 675 is blocked on the following code:

            // A merge was in progress. We need to retry the command after the merge
            // completes, as signaled by the closing of the replica's mergeComplete
            // channel. Note that the merge may have already completed, in which case
            // its mergeComplete channel will be nil.
            mergeCompleteCh := repl.getMergeCompleteCh()
            if mergeCompleteCh != nil {
                select {
                case <-mergeCompleteCh:
                    // Merge complete. Retry the command.
                case <-ctx.Done():
                    return nil, roachpb.NewError(ctx.Err())
                case <-s.stopper.ShouldQuiesce():
                    return nil, roachpb.NewError(&roachpb.NodeUnavailableError{})
                }
            }

So definitely seems to be an @benesch issue. This is sort of interesting because the merge queue is disabled at this point.

benesch commented 6 years ago

I’m hopeful this is the issue that’s fixed in https://github.com/cockroachdb/cockroach/pull/29324. Unfortunately I’m on the road right now so can’t fix. Apologies for the disruption. Mind plumbing in a t.Skip for now?

Not surprising to see the issue crop up here. This test manually turns on the merge queue.

On Thu, Aug 30, 2018 at 10:31 AM Peter Mattis notifications@github.com wrote:

Goroutine 675 is blocked on the following code:

      // A merge was in progress. We need to retry the command after the merge
      // completes, as signaled by the closing of the replica's mergeComplete
      // channel. Note that the merge may have already completed, in which case
      // its mergeComplete channel will be nil.
      mergeCompleteCh := repl.getMergeCompleteCh()
      if mergeCompleteCh != nil {
          select {
          case <-mergeCompleteCh:
              // Merge complete. Retry the command.
          case <-ctx.Done():
              return nil, roachpb.NewError(ctx.Err())
          case <-s.stopper.ShouldQuiesce():
              return nil, roachpb.NewError(&roachpb.NodeUnavailableError{})
          }
      }

So definitely seems to be an @benesch https://github.com/benesch issue. This is sort of interesting because the merge queue is disabled at this point.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cockroachdb/cockroach/issues/29169#issuecomment-417340518, or mute the thread https://github.com/notifications/unsubscribe-auth/AA15INIbvUstPG2QwFGOwV4prxMnPIqvks5uV_c4gaJpZM4WO9Qw .

petermattis commented 6 years ago

Let me see if the failure reproduces on #29324...nope.

I'll send a PR to skip this test for now. You can re-enable it in #29324.