search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 489 forks source link

Local reads via Stale Reads do not work #4627

Open Tema opened 2 years ago

Tema commented 2 years ago

Bug Report

What version of TiDB Operator are you using? v1.3.5 TidbCluster kind v6.1.0

What did you do? I try to perform local reads using Stale Reads described in https://docs.pingcap.com/tidb/stable/three-dc-local-read#perform-local-read-using-stale-read, but TIDB sends commands to all tikv across all zones.

What did you expect to see? Per documentation it should send requests to TiKV in the same zone

I use the following PD config:

      [replication]
        max-replicas = 3
        location-labels = ["topology.kubernetes.io/zone", "kubernetes.io/hostname"]

I also see that corresponding labels are assigned to TiKV nodes:

[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ab73507c-7tqz\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-b\"]"] [store_id=1]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ac1d383d-09th\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-c\"]"] [store_id=4]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-753c179c-tm45\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-a\"]"] [store_id=5]

However, I don't see same labels assigned to TiDB nodes, hence the feature does not work.

Ishiihara commented 2 years ago

@DanielZhangQD Can you help take a look?

Ishiihara commented 2 years ago

@Tema What are the labels for TiDB nodes?

Tema commented 2 years ago

@Tema What are the labels for TiDB nodes?

I don't see any. This is a problem. I expect operator to assign the same labels to TiDB as it does for TiKV nodes based on k8s "topology.kubernetes.io/zone" label (see log captured in the issue description) , so that we can leverage stale reads.

zimulala commented 2 years ago

@Tema What are the labels for TiDB nodes?

There is no documentation at present, please refer to the example in the test: https://github.com/pingcap/tidb/pull/19212/files#diff-373f5578b8f362a364b50b0d01919a57da64c128547ca10d49df8ae422237c82R215

In addition, how to configure the TiKV labels, refer to: https://docs.pingcap.com/tidb/stable/schedule-replicas-by-topology-labels#configure-labels-based-on-the-cluster-topology

DanielZhangQD commented 2 years ago

Adding label for TiDB Pods is not supported yet.

zhangjinpeng87 commented 2 years ago

After the following PR landed and released, this issue can be fixed. https://github.com/pingcap/tidb/pull/36845 https://github.com/pingcap/tidb-operator/pull/4663