zeromicro / go-zero

A cloud-native Go microservices framework with cli tool for productivity.
https://go-zero.dev
MIT License
29.2k stars 3.95k forks source link

ZRpc + Etcd + Docker Env, After the ZRpc client container slept for 1 hour, An ZRpc client request was initiated again, this request will exception and could not be recovered. The error "context deadline exceeded" appeared every 4 seconds in the log #4420

Open ysw-jingzhe opened 2 weeks ago

ysw-jingzhe commented 2 weeks ago

Describe the bug Zrpc + Etcd + Docker 容器微服务架构, Zrpc Client 端容器在刚启动时 rpc 之间请求正常,可以正常的通过ETCD发现服务端并请求到服务端数据。 但是在该Client 端容器睡眠1小时(1个小时内无任何外部请求) 后,再次触发Rpc客户端请求, 此时Rpc请求出现超时异常,且日志中每隔4秒出现一次 “error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240”。 此时检查 Docker 容器正常,容器中网络也正常。

备注:在不使用容器直接在服务器上运行该Client端的Go程序,就不会存在该问题; 在相同容器中使用 kratos 配套google.golang.org/grpc v1.46.2 不会存在该问题;

Zrpc+Etcd+Docker microservice Env, the Zrpc client container requests normally between RPC at the beginning of startup, and can discover the server and request server data through ETCD normally. However, after the client-side container had slept for 1 hour (with no external requests within the hour), the Rpc client request was triggered again. At this time, the Rpc request encountered a timeout exception and appeared every 4 seconds in the log "error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240".

Note: If the Go program on the client side is run directly on the server without using a container, this issue will not exist; Using Kratos + “google.golang.org/grpc v1.46.2” in the same container will not result in this issue;

Docker Image: alpine:latest or ubuntu 20.4 Etcd: etcd:v3.5.12 Zrpc: github.com/zeromicro/go-zero v1.7.2 Grpc: google.golang.org/grpc v1.67.0

To Reproduce Steps to reproduce the behavior, if applicable:

  1. The code is

    // client init
    conn = zrpc.MustNewClient(zrpc.RpcClientConf{
     Etcd: discov.EtcdConf{
        Hosts: ["127.0.0.1:2379"],
        Key: "examination.rpc"
     },
    })
    
    // client every request
    client := greet.NewGreetClient(conn.Conn())
    resp,err := client.Debug(ctx, in, opts...)
    
  2. The error is

    2024-10-11T09:33:40.108Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:44.109Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:48.111Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:52.113Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:56.115Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    

Expected behavior RPC requests are normal and stable

Screenshots

Environments (please complete the following information):

More description

zhoushuguang commented 2 weeks ago

I'll try to reproduce it.

ysw-jingzhe commented 2 weeks ago

thanks

ysw-jingzhe commented 1 week ago

Is there any progress? How to be stable? rpc for docker containers