search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.57k stars 2.83k forks source link

[Bug]: fail to Query on QueryNode 3: worker(3) query failed: getrandom #36271

Open liny1224 opened 1 week ago

liny1224 commented 1 week ago

Is there an existing issue for this?

Environment

- Milvus version:milvusdb/milvus:v2.4.5 
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):  2.4.6
- OS(Ubuntu or CentOS):  CentOS7
- CPU/Memory:   8/32G
- GPU: 
- Others:

Current Behavior

1:Start Docker using standalone_embed.sh, docker images milvusdb/milvus: v2.4.5. Start normally. It can be accessed through attu. 2:After creating and importing data, vector retrieval can be performed on the client. search. But Quey reported an error, and Attu also reported an error when viewing data. code: `

def search(self, query, limit=10, min_score=0.0):
        result = []
        embeddings = self.encode([query])
        query_vector = embeddings['dense']
        anns_field = 'materialTextDenseVector'
        group_by_field = 'materialId'
        data = self._vector_search(query_vector, limit, anns_field, group_by_field)
        return data
    def query(self, material_ids:List[str] = None, limit=10):
        filter = ''
        if material_ids:
            material_ids_str = ', '.join(f'"{mid}"' for mid in material_ids)
            filter = f'materialId in [{material_ids_str}]'

        output_fields =['materialId','originalContent','materialText','createTime','updateTime']

        return self.client.query(collection_name=self.collection_name, filter=filter, output_fields=output_fields,limit=limit)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2024/09/13 15:49:32.477 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=2e76020249af0c35894b7e72f0e63118] [collectionID=452523112655751125] [indexName=] [timestamp=0] [2024/09/13 15:49:32.477 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=452523112655751125] [indexID=452523112655751132] [totalRows=0] [indexRows=0] [pendingIndexRows=0] [state=Finished] [failReason=] [2024/09/13 15:49:32.477 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=2e76020249af0c35894b7e72f0e63118] [collectionID=452523112655751125] [indexName=] [2024/09/13 15:49:33.106 +00:00] [WARN] [segments/segment.go:633] ["CStatus returns err"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [partitionID=452523112655751126] [segmentID=452523112656952069] [msgID=452524098309849273] [segmentType=Growing] [error=getrandom] [extra="Retrieve failed"] [2024/09/13 15:49:33.106 +00:00] [WARN] [querynodev2/services.go:861] ["failed to query channel"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [msgID=452524098309849273] [collectionID=452523112655751125] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [scope=Streaming] [error=getrandom] [2024/09/13 15:49:33.106 +00:00] [WARN] [delegator/delegator.go:637] ["failed to execute sub task"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [replicaID=452523112814608385] [taskType=Query] [nodeID=3] [error="worker(3) query failed: getrandom"] [2024/09/13 15:49:33.106 +00:00] [WARN] [delegator/delegator.go:657] ["Delegator execute subTask failed"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [replicaID=452523112814608385] [taskType=Query] [error="worker(3) query failed: getrandom"] [2024/09/13 15:49:33.106 +00:00] [WARN] [delegator/delegator.go:506] ["Delegator query failed"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [replicaID=452523112814608385] [error="worker(3) query failed: getrandom"] [2024/09/13 15:49:33.106 +00:00] [WARN] [querynodev2/handlers.go:221] ["failed to query on delegator"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [msgID=452524098309849273] [collectionID=452523112655751125] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [scope=All] [error="worker(3) query failed: getrandom"] [2024/09/13 15:49:33.106 +00:00] [WARN] [proxy/task_query.go:557] ["QueryNode query result error"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collection=452523112655751125] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_0_452523112655751125v0] [errorCode=UnexpectedError] [reason="worker(3) query failed: getrandom"] [2024/09/13 15:49:33.106 +00:00] [WARN] [proxy/lb_policy.go:182] ["search/query channel failed"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [collectionName=jsdp_material] [channelName=by-dev-rootcoord-dml_0_452523112655751125v0] [nodeID=3] [error="fail to Query on QueryNode 3: worker(3) query failed: getrandom"] [errorVerbose="fail to Query on QueryNode 3: worker(3) query failed: getrandom\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).queryShard\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:558\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).Execute.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) fail to Query on QueryNode 3\nWraps: (3) worker(3) query failed: getrandom\nError types: (1) withstack.withStack (2) errutil.withPrefix (3) merr.milvusError"] [2024/09/13 15:49:33.107 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [retried=0] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom"] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:188\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).queryShard\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:558\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).Execute.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (4) fail to Query on QueryNode 3\nWraps: (5) worker(3) query failed: getrandom\nError types: (1) withstack.withStack (2) errutil.withPrefix (3) withstack.withStack (4) errutil.withPrefix (5) merr.milvusError"] [2024/09/13 15:49:33.307 +00:00] [INFO] [proxy/meta_cache.go:997] ["clearing shard cache for collection"] [collectionName=jsdp_material] [2024/09/13 15:49:33.308 +00:00] [WARN] [proxy/lb_policy.go:126] ["no available shard delegator found"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [collectionName=jsdp_material] [channelName=by-dev-rootcoord-dml_0_452523112655751125v0] [nodes="[3]"] [excluded="[3]"] [2024/09/13 15:49:33.308 +00:00] [WARN] [proxy/lb_policy.go:157] ["failed to select node for shard"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collectionID=452523112655751125] [collectionName=jsdp_material] [channelName=by-dev-rootcoord-dml_0_452523112655751125v0] [nodeID=-1] [error="channel not available[channel=no available shard delegator found]"] [2024/09/13 15:49:33.709 +00:00] [WARN] [proxy/task_query.go:462] ["fail to execute query"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [collection=452523112655751125] [partitionIDs="[]"] [requestType=query] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom"] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:188\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).queryShard\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:558\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).Execute.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (4) fail to Query on QueryNode 3\nWraps: (5) worker(3) query failed: getrandom\nError types: (1) withstack.withStack (2) errutil.withPrefix (3) withstack.withStack (4) errutil.withPrefix (5) merr.milvusError"] [2024/09/13 15:49:33.709 +00:00] [WARN] [proxy/task_scheduler.go:469] ["Failed to execute task: "] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [error="failed to query: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom"] [errorVerbose="failed to query: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:463\n | github.com/milvus-io/milvus/internal/proxy.(taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:466\n | github.com/milvus-io/milvus/internal/proxy.(taskScheduler).queryLoop.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:545\n | github.com/milvus-io/milvus/pkg/util/conc.(Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\nWraps: (2) failed to query\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:188\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).queryShard\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:558\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).Execute.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (6) fail to Query on QueryNode 3\nWraps: (7) worker(3) query failed: getrandom\nError types: (1) withstack.withStack (2) errutil.withPrefix (3) withstack.withStack (4) errutil.withPrefix (5) withstack.withStack (6) errutil.withPrefix (7) merr.milvusError"] [2024/09/13 15:49:33.709 +00:00] [WARN] [proxy/impl.go:3505] ["Query failed to WaitToFinish"] [traceID=6caac67083d6faad7a8cc7e9d91101c9] [role=proxy] [db=default] [collection=jsdp_material] [partitions="[]"] [useDefaultConsistency=false] [error="failed to query: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom"] [errorVerbose="failed to query: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0: fail to Query on QueryNode 3: worker(3) query failed: getrandom\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).Execute\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:463\n | github.com/milvus-io/milvus/internal/proxy.(taskScheduler).processTask\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:466\n | github.com/milvus-io/milvus/internal/proxy.(taskScheduler).queryLoop.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:545\n | github.com/milvus-io/milvus/pkg/util/conc.(Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\nWraps: (2) failed to query\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:188\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_0_452523112655751125v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(queryTask).queryShard\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:558\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:180\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).ExecuteWithRetry\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n | github.com/milvus-io/milvus/internal/proxy.(LBPolicyImpl).Execute.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:218\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (6) fail to Query on QueryNode 3\nWraps: (7) worker(3) query failed: getrandom\nError types: (1) withstack.withStack (2) errutil.withPrefix (3) withstack.withStack (4) errutil.withPrefix (5) withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"] [2024/09/13 15:49:34.683 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=f3e849e6685e607bd0d081b6b817ca6c] [collectionID=452523112655751125]

Anything else?

No response

yanliang567 commented 1 week ago

i guess it was caused by an invalid filter in the query request. as the param material_ids is a list already, please try

def query(self, material_ids:List[str] = None, limit=10):
        filter = f'materialId in {material_ids}'
        output_fields =['materialId','originalContent','materialText','createTime','updateTime']
        return self.client.query(collection_name=self.collection_name, filter=filter, output_fields=output_fields,limit=limit)

/assign @liny1224 /unassign