pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.28k stars 5.84k forks source link

encountered TSO stream being closed unexpectedly #51328

Open aytrack opened 8 months ago

aytrack commented 8 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. deploy a v7.5.0 cluster
  2. enable dist task and add index
  3. during add index and update cluster to v7.5.1

2. What did you expect to see? (Required)

add index success

3. What did you see instead (Required)

get error for add index

Error 1105 (HY000): encountered TSO stream being closed unexpectedly

the tidb log

2024/02/26 18:35:39.713 +08:00] [ERROR] [pd_service_discovery.go:284] ["[pd] failed to update service mode"] [urls="[http://pd-peer:2379]"] [error="[PD:client:ErrClientGetClusterInfo]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.200.59.224:2379: connect: connection refused\" target:pd-peer:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.200.59.224:2379: connect: connection refused\" target:pd-peer:2379 status:TRANSIENT_FAILURE"]
[2024/02/26 18:35:40.085 +08:00] [ERROR] [tso_dispatcher.go:493] ["[tso] getTS error"] [dc-location=global] [stream-addr=http://pd-peer:2379] [error="[PD:client:ErrClientGetTSO]get TSO failed, after processing requests"]
[2024/02/26 18:35:40.085 +08:00] [ERROR] [pd.go:236] ["updateTS error"] [txnScope=global] [error="[PD:client:ErrClientTSOStreamClosed]encountered TSO stream being closed unexpectedly"]

releated issue: #47291

4. What is your TiDB version? (Required)

v7.5.0 -> v7.5.1

lance6716 commented 8 months ago

@CabinfeverB :smile: