search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.14k stars 5.83k forks source link

The variable tidb_allow_fallback_to_tikv can't take effect when the tiflash nodes stop #24421

Closed lilinghai closed 3 years ago

lilinghai commented 3 years ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tpcc 10k warehouses and tiflash replica 2
tiup cluster stop xxx -R tiflash
select sum(s_order_cnt) from stock where s_i_id=1; -- ERROR 9012 (HY000): TiFlash server timeout
set @@tidb_allow_fallback_to_tikv="tiflash";
select sum(s_order_cnt) from stock where s_i_id=1; -- ERROR 9005 (HY000): Region is unavailable

The plan is

+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
| id                             | estRows       | task              | access object | operator info                                                 |
+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
| StreamAgg_27                   | 1.00          | root              |               | funcs:sum(Column#21)->Column#19                               |
| └─TableReader_28               | 1.00          | root              |               | data:StreamAgg_9                                              |
|   └─StreamAgg_9                | 1.00          | batchCop[tiflash] |               | funcs:sum(Column#22)->Column#21                               |
|     └─Projection_35            | 10064.41      | batchCop[tiflash] |               | cast(tpcc.stock.s_order_cnt, decimal(32,0) BINARY)->Column#22 |
|       └─Selection_26           | 10064.41      | batchCop[tiflash] |               | eq(tpcc.stock.s_i_id, 1)                                      |
|         └─TableFullScan_25     | 1000000000.00 | batchCop[tiflash] | table:stock   | keep order:false                                              |
+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
6 rows in set (0.05 sec)

2. What did you expect to see? (Required)

The last query fallback to tikv

3. What did you see instead (Required)

ERROR 9005 (HY000): Region is unavailable

4. What is your TiDB version? (Required)

Release Version: v4.0.0-beta.2-2791-g207ce344c-dirty Edition: Community Git Commit Hash: 207ce344cbb044ffb1b2681f1ba320a154979f6d Git Branch: master UTC Build Time: 2021-05-02 15:24:42 GoVersion: go1.13 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false

winoros commented 3 years ago

Is the environment available? Or a more detailed log stack? @lilinghai

lilinghai commented 3 years ago

The second query log is

[2021/05/06 12:12:50.161 +00:00] [WARN] [backoff.go:332] ["regionMiss backoffer.maxSleep 10000ms is exceeded, errors:\nCannot find region with TiFlash peer at 2021-05-06T12:12:49.038191558Z\nCannot find region with TiFlash peer at 2021-05-06T12:12:49.601745515Z\nCannot find region with TiFlash peer at 2021-05-06T12:12:50.161559434Z"]
[2021/05/06 12:12:50.162 +00:00] [INFO] [conn.go:878] ["command dispatched failed"] [conn=57] [connInfo="id:57, addr:100.64.10.10:56638 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select count(*) from stock"] [txn_mode=PESSIMISTIC] [err="[tikv:9005]Region is unavailable\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/tidb_master/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/home/jenkins/agent/workspace/tidb_master/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/copr.buildBatchCopTasks\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:155\ngithub.com/pingcap/tidb/store/copr.(*CopClient).sendBatch\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:181\ngithub.com/pingcap/tidb/store/copr.(*CopClient).Send\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:73\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/distsql/distsql.go:77\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/distsql/distsql.go:126\ngithub.com/pingcap/tidb/executor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).buildResp\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:241\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:157\ngithub.com/pingcap/tidb/executor.(*baseExecutor).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/executor.go:163\ngithub.com/pingcap/tidb/executor.(*StreamAggExec).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/aggregate.go:1104\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/adapter.go:336\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/session/session.go:1576\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/session/session.go:1471\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/driver_tidb.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1697\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1569\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1103\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:861\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/server.go:481\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

I will offer the env to you with other chat tools.

ichn-hu commented 3 years ago

/unlabel sig/execution

ichn-hu commented 3 years ago

/label sig/planner

ti-chi-bot commented 3 years ago

@ichn-hu: The label(s) sig/planner cannot be applied. These labels are supported: CHANGELOG, challenge-program, compatibility-breaker, contribution, correctness, first-time-contributor, good-first-issue, hacktoberfest, high-performance, hptc, need-more-info, needs-cherry-pick-3.0, needs-cherry-pick-3.1, needs-cherry-pick-4.0, needs-cherry-pick-5.0-rc, needs-cherry-pick-5.0, proposal, release-note, require-LGT3, security, tools.

In response to [this](https://github.com/pingcap/tidb/issues/24421#issuecomment-836302631): >/label sig/planner Instructions for interacting with me using PR comments are available [here](https://prow.tidb.io/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
ti-srebot commented 3 years ago

Please edit this comment or add a new comment to complete the following information

Not a bug

  1. Remove the 'type/bug' label
  2. Add notes to indicate why it is not a bug

Duplicate bug

  1. Add the 'type/duplicate' label
  2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added Example for how to fill out the template: https://github.com/pingcap/tidb/issues/20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions