zeromicro / go-zero

A cloud-native Go microservices framework with cli tool for productivity.
https://go-zero.dev
MIT License
29.31k stars 3.96k forks source link

Infinite output of ETCD logs, causing service crashes #3823

Open bCoder778 opened 10 months ago

bCoder778 commented 10 months ago

Describe the bug A large number of identical error logs without any pauses result in 100% CPU usage. The log files exceeded a few gigabytes in a short amount of time, causing the server to crash.

To Reproduce

The program started working normally, but after running for a period of time, it began to output the same logs without restrictions, causing the server to crash

  1. The code is

    Etcd: Hosts:

    • ...:379 Key: .rpc User: user Pass:
  2. The error is

    "etcd monitor chan has been canceled, error: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}

Environments (please complete the following information):

codeErrorSleep commented 10 months ago

We have also encountered this problem. You can try to downgrade the version of gRPC a bit; that's how we resolved it.

bCoder778 commented 10 months ago

We have also encountered this problem. You can try to downgrade the version of gRPC a bit; that's how we resolved it.

Thank you very much. Can you tell me which specific version of GRPC you are using

arllen133 commented 4 months ago

Has it been resolved yet?

iyyzh commented 1 month ago

mark