search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.21k stars 3.82k forks source link

sql: TestReacquireLeaseOnRestart failed #136361

Open cockroach-teamcity opened 5 days ago

cockroach-teamcity commented 5 days ago

sql.TestReacquireLeaseOnRestart failed on release-24.1.8-rc @ 8450f1eef817abf295de10517cf0720e65e7a9b0:

      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2573 +0xdf6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2173 +0x8c4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1722 +0x1030
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1241 +0xad9
  github.com/cockroachdb/cockroach/pkg/kv.lookupRangeFwdScan()
      github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:326 +0x841
  github.com/cockroachdb/cockroach/pkg/kv.RangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:201 +0x3e9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:963 +0x546
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.(*RangeCache).performRangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:975 +0x112
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.tryLookupImpl.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:869 +0xc4
  github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
      github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:28 +0x12d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.tryLookupImpl()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:866 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.(*RangeCache).tryLookup.func3()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:764 +0xa4
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:384 +0x4b
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:315 +0x185
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:383 +0x2c4
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan.gowrap1()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:353 +0xc6

Goroutine 110073 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1089 +0x92f
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:871 +0xe04
  github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:170 +0x1d7
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:90 +0x561
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux-fm()
      <autogenerated>:1 +0xc4
  github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1909 +0x3b5
  github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1()
      github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:186 +0x1c6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x262
==================

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #129421 sql: TestReacquireLeaseOnRestart failed [C-test-failure O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

Jira issue: CRDB-44995

yuzefovich commented 21 hours ago

On a quick glance, it seems like due to a test problem.

WARNING: DATA RACE
Read at 0x00c00c9f2f50 by goroutine 108472:
  reflect.Value.IsNil()
      GOROOT/src/reflect/value.go:1572 +0xd3
  encoding/json.sliceEncoder.encode()
      GOROOT/src/encoding/json/encode.go:801 +0xf4
  encoding/json.sliceEncoder.encode-fm()
      <autogenerated>:1 +0x84
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:704 +0x2bd
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xe4
  encoding/json.ptrEncoder.encode()
      GOROOT/src/encoding/json/encode.go:876 +0x3f1
  encoding/json.ptrEncoder.encode-fm()
      <autogenerated>:1 +0x84
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:704 +0x2bd
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xe4
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:704 +0x2bd
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xe4
  encoding/json.ptrEncoder.encode()
      GOROOT/src/encoding/json/encode.go:876 +0x3f1
  encoding/json.ptrEncoder.encode-fm()
      <autogenerated>:1 +0x84
  encoding/json.(*encodeState).reflectValue()
      GOROOT/src/encoding/json/encode.go:321 +0x83
  encoding/json.(*encodeState).marshal()
      GOROOT/src/encoding/json/encode.go:297 +0xea
  encoding/json.(*Encoder).Encode()
      GOROOT/src/encoding/json/stream.go:209 +0x16f
  github.com/cockroachdb/cockroach/pkg/server.NewServer.GRPCTransportFactory.func23.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:114 +0x20b
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x262
Previous write at 0x00c00c9f2f50 by goroutine 110073:
  github.com/cockroachdb/cockroach/pkg/roachpb.(*Transaction).UpdateObservedTimestamp()
      github.com/cockroachdb/cockroach/pkg/roachpb/pkg/roachpb/data.go:1450 +0x2e4
  github.com/cockroachdb/cockroach/pkg/sql_test.TestReacquireLeaseOnRestart.func1()
      github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/txn_restart_test.go:1178 +0x374
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes()
      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:197 +0x833
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes()
      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:188 +0xd11
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes()
      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:198 +0xde
  github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
      github.com/cockroachdb/cockroach/pkg/server/node.go:1528 +0x930
  github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
      github.com/cockroachdb/cockroach/pkg/server/node.go:1723 +0x4a4
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:703 +0x74
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
      github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92 +0x72b
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:165 +0xf0
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:90 +0x2b6
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
      <autogenerated>:1 +0xd6
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:132 +0x6a
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:332 +0x185
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:130 +0x150
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:713 +0x75
  github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
      github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:223 +0x5ed
  github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:898 +0x199
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:783 +0x462
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
      github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:906 +0xf2
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
      <autogenerated>:1 +0x29
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:209 +0x23e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:187 +0x139
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:71 +0x2ec
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x6b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2726 +0x26db
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2173 +0x8c4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1722 +0x1030
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1241 +0xad9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x26b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41 +0x1d9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:234 +0x272
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshAndRetrySend()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:351 +0x6f4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:286 +0x61d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshAndRetrySend()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:351 +0x6f4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:286 +0x61d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshAndRetrySend()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:351 +0x6f4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:286 +0x61d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162 +0x1d8
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144 +0x95e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:321 +0x244
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112 +0x3d8
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:241 +0x7bb
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:532 +0xa76
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:1133 +0x181
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
      github.com/cockroachdb/cockroach/pkg/kv/txn.go:1286 +0x37c
  github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeTxnKVFetcherDefaultSendFunc.func2()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:274 +0x55
  github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:578 +0x9e8
  github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:861 +0x1f68
  github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch-fm()
      <autogenerated>:1 +0xb8
  github.com/cockroachdb/cockroach/pkg/sql/row.(*kvBatchFetcherHelper).NextBatch()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:1010 +0xcc
  github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch()
      <autogenerated>:1 +0xb9
  github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:279 +0x15c
  github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV()
      github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:300 +0x8f
  github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch()
      github.com/cockroachdb/cockroach/pkg/sql/colfetcher/cfetcher.go:688 +0x17c
  github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next()
      github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_scan.go:238 +0x93
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
      github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 +0x6b
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next()
      github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:54 +0x44
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
      github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 +0x6b
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:113 +0x5b
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next-fm()
      <autogenerated>:1 +0x1f
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:145 +0x7c
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:121 +0x95
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Next()
      <autogenerated>:1 +0x31
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245 +0x63
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter-fm()
      <autogenerated>:1 +0x1f
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:145 +0x7c
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:249 +0x28a
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:281 +0x23b
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:315 +0x352
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:923 +0x1815
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1985 +0x32e
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1699 +0x1b0
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1702 +0x353
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2462 +0xb88
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2012 +0x23ec
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1158 +0x8ff3
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:140 +0x184
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3465 +0x539
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:139 +0xbbd
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2287 +0x94b
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2292 +0x1392
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2209 +0x3ea
  github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:960 +0x216
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:251 +0x6c6
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func3()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1092 +0x21c
Goroutine 108472 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x69c
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0xaf
  github.com/cockroachdb/cockroach/pkg/server.NewServer.GRPCTransportFactory.func23()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:92 +0x12c
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2573 +0xdf6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2173 +0x8c4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1722 +0x1030
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1241 +0xad9
  github.com/cockroachdb/cockroach/pkg/kv.lookupRangeFwdScan()
      github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:326 +0x841
  github.com/cockroachdb/cockroach/pkg/kv.RangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:201 +0x3e9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:963 +0x546
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.(*RangeCache).performRangeLookup()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:975 +0x112
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.tryLookupImpl.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:869 +0xc4
  github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
      github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:28 +0x12d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.tryLookupImpl()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:866 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache.(*RangeCache).tryLookup.func3()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:764 +0xa4
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:384 +0x4b
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:315 +0x185
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:383 +0x2c4
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan.gowrap1()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:353 +0xc6
Goroutine 110073 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1089 +0x92f
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:871 +0xe04
  github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:170 +0x1d7
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:90 +0x561
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux-fm()
      <autogenerated>:1 +0xc4
  github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1909 +0x3b5
  github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1()
      github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:186 +0x1c6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x262