search

tikv / client-go

Go client for TiKV
Apache License 2.0
281 stars 220 forks source link

The metric `tidb_tikvclient_batch_recv_latency` is not reasonable #539

Open you06 opened 2 years ago

you06 commented 2 years ago

Found by @zyguan

Start a cluster and wait a while, then run some workload, we got "Batch Receive Average Duration" panel like this.

image

The way we observe the receiving time is not reasonable. In an idle cluster, this function is waiting for responses a long time while there are no in-fly requests, so the receiving time is high. In a busy cluster, the responses are pending in the client's buffer and we just loop reading them from memory, so the receiving time is extremely low(avg 547us).

https://github.com/tikv/client-go/blob/79a0fc59e4d5012095282344f5fdbca1492de771/internal/client/client_batch.go#L427-L449

disksing commented 2 years ago

maybe change it to another title? such as 'Batch Wait for response duration'?