search

milvus-io / milvus-operator

milvus-operator
Apache License 2.0
62 stars 30 forks source link

Etcd Client Sends Invalid Authority Header #260

Open jeff-hegerhorst-deepsee opened 1 year ago

jeff-hegerhorst-deepsee commented 1 year ago

Overview Operator Image Version: milvusdb/milvus-operator:v0.7.13

The etcd client being used by the operator is sending an invalid authority header when trying to connect to etcd. This causes issues when etcd is being hosted behind a proxy like Istio because the sidecar rejects the request. This results in the health check of the etcd cluster to fail with the following logs:

{"level":"warn","ts":"2023-06-27T14:48:14.903Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":0,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:15.933Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":1,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:16.963Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":2,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:17.993Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":3,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:18.899Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":4,"error":"rpc error: code = DeadlineExceed
ed desc = context deadline exceeded"}
2023/06/27 14:48:18 condition not ready EtcdReady False EtcdNotReady All etcd endpoints are unhealthy:[ds-milvus-etcd.default:2379:context deadlin
e exceeded]

Solution The etcd client for the operator needs to be updated to a version greater than 3.5.0 see https://github.com/etcd-io/etcd/issues/13192

haorenfsa commented 1 year ago

We changed default etcd version to 3.5.5