milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
28.13k stars 2.71k forks source link

[Bug]: proxy warning log ["status is nil, please fix it"] for query node #34171

Open hwc1824 opened 1 week ago

hwc1824 commented 1 week ago

Is there an existing issue for this?

Environment

- Milvus version: v2.4.4
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): v2.4.0
- OS(Ubuntu or CentOS):  
- CPU/Memory: 
- GPU: 
- Others: Hosted in K8s

Current Behavior

Though searching on collections, loading / releasing collection are working fine, all the proxy pod showed this warning message repeatedly, I tried to rolling restart the proxy and querynode but it didn't get resolved.

[2024/06/25 09:53:36.428 +00:00] [WARN] [grpcclient/client.go:520] ["status is nil, please fix it"] [traceID=81b39ecc6873a390a9c849cbdfd0efb2] [client_role=querynode-323] 
[stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2
    /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:520
github.com/milvus-io/milvus/pkg/util/retry.Handle
    /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:104
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call
    /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:467
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
    /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:554
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
    /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570
github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
    /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90
github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search
    /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:202
github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard
    /go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:685
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1
    /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180
github.com/milvus-io/milvus/pkg/util/retry.Do
    /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry
    /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154
github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2
    /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218
golang.org/x/sync/errgroup.(*Group).Go.func1
    /go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

I didn't found the similar warning message when I lower the milvus version to v2.4.0

xiaofan-luan commented 1 week ago

/assign @weiliu1031

xiaofan-luan commented 1 week ago

/assign @aoiasd

smellthemoon commented 1 week ago

may related with #34100