8-RESOURCE_EXHAUSTED, message: "Received message larger than max (9612720 vs. 4194304)

[abelard2008]

Based on 3 TiKV nodes + 3 PD nodes, I deployed 1 Tidis node, and the part of config.toml

[server]
pd_addrs = "172.26.9.81:2379,172.26.9.82:2379,172.26.9.83:2379"
instance_id = "1"
prometheus_listen = "0.0.0.0"
prometheus_port = 8080
log_level = "info"
log_file = "tikv-service.log"

After some writing (Redis) sorted set, I got some error message in tikv-service.log:

2023/01/04 12:54:40.182 +08:00 ERRO error occured so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message larger than max (9612720 vs. 4194304)
2023/01/04 12:54:40.183 +08:00 ERRO connection error TikvClient(Grpc(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Received message larger than max (9612720 vs. 4194304)", details: [] })))
2023/01/04 12:54:44.090 +08:00 ERRO error occured so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message larger than max (9723284 vs. 4194304)

[yongman]

The error message show like the size of message is exceed the grpc limitation. Can you provide more details infomation about the commands and the datasets?

[abelard2008]

I only used Redis cmd: zadd insert(based on hiredis), and zrangebyscore/zrem (python). I will insert 10,000,000 records into the sorted set named filepath-sorted-set, but when tidis server crashed, only tens of thousands of records were inserted into the sorted set.

PS: my Tidis server can't restart now. When I run tidis-server --config config.toml my tikv-service.log start to output repeatly:

2023/01/04 18:26:18.232 +08:00 INFO connect to tikv endpoint: "172.26.9.83:20160"
2023/01/04 18:26:18.237 +08:00 INFO connect to tikv endpoint: "172.26.9.81:20160"
2023/01/04 18:26:18.244 +08:00 INFO connect to tikv endpoint: "172.26.9.82:20160"
2023/01/04 18:26:18.987 +08:00 ERRO error occured so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message l
arger than max (72327769 vs. 4194304)
2023/01/04 18:26:18.988 +08:00 ERRO connection error TikvClient(Grpc(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Received
 message larger than max (72327769 vs. 4194304)", details: [] })))
2023/01/04 18:26:24.779 +08:00 ERRO error occured so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message l
arger than max (72327769 vs. 4194304)
2023/01/04 18:26:24.779 +08:00 ERRO connection error TikvClient(Grpc(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Received
 message larger than max (72327769 vs. 4194304)", details: [] })))