search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.37k stars 2.82k forks source link

[Bug]: vector output fields for IVF_FLAT index is not allowed #26960

Closed stevetracvc closed 10 months ago

stevetracvc commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version: docker milvusdb/milvus:2.3.0-latest downloaded Sept 8
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.0
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 16G (AWS EC2 instance)
- GPU: N/A
- Others:

Current Behavior

If I insert more than 40,000 records, then I can no longer query and request the embedding vectors. I instead get the following error in my python script:

RPC error: [query], <MilvusException: (code=1, message=failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable)>, <Time:{'RPC start': '2023-09-09 07:45:03.329872', 'RPC error': '2023-09-09 07:45:04.449479'}> Traceback (most recent call last):

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py:473 in exec_code exec_fun(compile(ast_code, filename, 'exec'), ns_globals, ns_locals)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/spyder_kernels/py3compat.py:356 in compat_exec exec(code, globals, locals)

File /mnt/TRAC/TRAC-confidential-ML/machine-learning/PB-timeseries/industry_embeddings/milvus_test.py:94 res = collection.query(

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/orm/collection.py:912 in query return conn.query(

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:127 in handler raise e from e

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:123 in handler return func(*args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:162 in handler return func(self, *args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:102 in handler raise e from e

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:68 in handler return func(*args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:1350 in query raise MilvusException(response.status.error_code, response.status.reason)

MilvusException: <MilvusException: (code=1, message=failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable)>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/spyder_kernels/py3compat.py:356 in compat_exec exec(code, globals, locals)

File /mnt/TRAC/TRAC-confidential-ML/machine-learning/PB-timeseries/industry_embeddings/milvus_test.py:94 res = collection.query(

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/orm/collection.py:912 in query return conn.query(

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:127 in handler raise e from e

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:123 in handler return func(*args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:162 in handler return func(self, *args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:102 in handler raise e from e

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/decorators.py:68 in handler return func(*args, **kwargs)

File ~/miniconda3/envs/milvus/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:1350 in query raise MilvusException(response.status.error_code, response.status.reason)

MilvusException: <MilvusException: (code=1, message=failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable)>

Expected Behavior

Once I insert vectors, I should be able to retrieve them at a later date. Maybe I'm missing the point of how this software is supposed to work. I saw #24822 and #24409 which are both closed, but I don't understand the reasoning.

Currently, you cannot search the vectors without providing a vector. I've seen that you're working on allowing vector searches by primary key (ie, specify the primary key in the search rather than the query, so that the database will query the vector and then do the search, all on the server side). But this should imply that it currently is possible to retrieve a vector from the database (eg, query by primary key), then feed it back to the database for a vector search. Even the tutorials show that this is supposed to be possible, and indeed, it works until you insert too many records. https://milvus.io/docs/v2.0.x/query.md shows that boon_intro, a vector, can be retrieved.

Steps To Reproduce

the following code is pieced together from the official tutorial pages, my only change is increasing the number of records created and inserted. If you change the number of insert iterations to 2 in the insert_data section, then the final query works. If you put it at 3 or larger, it doesn't work. I've also tried with different numbers of shards and that doesn't seem to change anything.


#%% build_schema

from pymilvus import CollectionSchema, FieldSchema, DataType
dims_intro = 768
record_count = 400_000
book_id = FieldSchema(
  name="book_id",
  dtype=DataType.INT64,
  is_primary=True,
)
word_count = FieldSchema(
  name="word_count",
  dtype=DataType.INT64,
)
book_intro = FieldSchema(
  name="book_intro",
  dtype=DataType.FLOAT_VECTOR,
  dim=dims_intro,
)
schema = CollectionSchema(
  fields=[book_id, word_count, book_intro],
  description="Test book search"
)
collection_name = "book"

#%% create_collection

from pymilvus import Collection
collection = Collection(
    name=collection_name,
    schema=schema,
    # shards_num=1,
    shards_num=2,
    consistency_level="Strong"
    )

#%% create_data

import random
chunks = 20  # can't insert 400k all at once anyway

data = []
for n in range(chunks):
    d = [
      [n * record_count // chunks + i for i in range(record_count // chunks)],
      [i for i in range(10_000, 10_000 + record_count // chunks)],
      [[random.random() for _ in range(dims_intro)] for _ in range(record_count // chunks)],
    ]
    data.append(d)

#%% insert_data

from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
[collection.insert(data[i]) for i in range(
    # chunks # doesn't work
    3 # doesn't work
    # 2 # works, 40,000 rows
    )]

# not listed in tutorials
collection.flush()

#%% create_index

index_params = {
  "metric_type":"COSINE",
  "index_type":"IVF_FLAT",
  "params":{"nlist":1024}
}
from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
collection.create_index(
  field_name="book_intro",
  index_params=index_params,
  index_name="idx_book_intro"
)

#%% query

from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
collection.load()

res = collection.query(
  expr = "book_id in [2,4,6,8]",
  output_fields = ["book_id", "book_intro"],
  consistency_level="Strong"
)


### Milvus Log

milvus-standalone | [2023/09/09 14:08:14.780 +00:00] [INFO] [rootcoord/root_coord.go:1101] ["received request to describe collection"] [traceID=af9e4ac0fe585aeb7f63c4c64d9db798] [collectionName=book] [dbName=tra
c_data] [id=0] [ts=18446744073709551615] [allowUnavailable=false]
milvus-standalone | [2023/09/09 14:08:14.780 +00:00] [INFO] [rootcoord/root_coord.go:1130] ["done to describe collection"] [traceID=af9e4ac0fe585aeb7f63c4c64d9db798] [collectionName=book] [dbName=trac_data] [id=
0] [ts=18446744073709551615] [allowUnavailable=false] [collection_id=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.840 +00:00] [INFO] [observers/collection_observer.go:122] ["observes partitions timeout"] [partitionNum=1]
milvus-standalone | [2023/09/09 14:08:14.840 +00:00] [INFO] [observers/collection_observer.go:159] ["observe partitions status"] [partitionNum=1]
milvus-standalone | [2023/09/09 14:08:14.867 +00:00] [INFO] [rootcoord/root_coord.go:1101] ["received request to describe collection"] [traceID=c2a0237d585ea3247dc2b0e99a3c11f4] [collectionName=book] [dbName=tra
c_data] [id=0] [ts=18446744073709551615] [allowUnavailable=false]
milvus-standalone | [2023/09/09 14:08:14.867 +00:00] [INFO] [rootcoord/root_coord.go:1130] ["done to describe collection"] [traceID=c2a0237d585ea3247dc2b0e99a3c11f4] [collectionName=book] [dbName=trac_data] [id=
0] [ts=18446744073709551615] [allowUnavailable=false] [collection_id=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.953 +00:00] [INFO] [datacoord/index_service.go:521] ["receive DescribeIndex request"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440] [indexN
ame=] [timestamp=0]
milvus-standalone | [2023/09/09 14:08:14.953 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444129503507318440] [indexID=444129503507519148] [totalRows=60000] [indexR
ows=60000] [pendingIndexRows=0] [state=Finished] [failReason=]
milvus-standalone | [2023/09/09 14:08:14.953 +00:00] [INFO] [datacoord/index_service.go:575] ["DescribeIndex success"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440] [indexName=]
milvus-standalone | [2023/09/09 14:08:14.954 +00:00] [INFO] [querycoordv2/services.go:199] ["load collection request received"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440] [repli
caNumber=1] [resourceGroups="[]"] [refreshMode=false] [schema="name:\"book\" description:\"Test book search\" fields:<fieldID:100 name:\"book_id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\
"word_count\" data_type:Int64 > fields:<fieldID:102 name:\"book_intro\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > "] [fieldIndexes="[444129503507519148]"]
milvus-standalone | [2023/09/09 14:08:14.954 +00:00] [INFO] [job/scheduler.go:162] ["start to pre-execute job"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.954 +00:00] [INFO] [job/scheduler.go:170] ["start to execute job"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.955 +00:00] [INFO] [meta/failed_load_cache.go:107] ["FailedLoadCache removes cache"] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.955 +00:00] [INFO] [rootcoord/root_coord.go:1420] ["received request to show partitions"] [traceID=cf65086a2efc624465d9e04872d3153a] [collection=] [collection_id=44412950
3507318440] [partitions="[]"] [allowUnavailable=false]
milvus-standalone | [2023/09/09 14:08:14.955 +00:00] [INFO] [rootcoord/root_coord.go:1451] ["done to show partitions"] [traceID=cf65086a2efc624465d9e04872d3153a] [collection=] [collection_id=444129503507318440] 
[partitions="[]"] [allowUnavailable=false] [partitions="[_default]"]
milvus-standalone | [2023/09/09 14:08:14.955 +00:00] [INFO] [job/scheduler.go:156] ["start to post-execute job"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:14.955 +00:00] [INFO] [job/scheduler.go:158] ["job finished"] [traceID=cf65086a2efc624465d9e04872d3153a] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:15.039 +00:00] [INFO] [querycoordv2/services.go:58] ["show collections request received"] [traceID=5e6312e2052db11883c4a1f6e5e02413] [collections="[444129503507318440]"]
milvus-standalone | [2023/09/09 14:08:15.128 +00:00] [INFO] [proxy/meta_cache.go:791] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [colle
ctionName=book] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:15.128 +00:00] [INFO] [querycoordv2/services.go:840] ["get shard leaders request received"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:15.462 +00:00] [WARN] [segments/segment.go:488] ["validate fields failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [partitionID=44412950
3507318441] [segmentID=444129503507518684] [error="vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.462 +00:00] [WARN] [querynodev2/services.go:919] ["failed to query channel"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [msgID=444142320382377985] [collectionID=444129503
507318440] [channel=by-dev-rootcoord-dml_8_444129503507318440v1] [scope=Historical] [error="vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.462 +00:00] [WARN] [delegator/delegator.go:451] ["failed to execute sub task"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel=by-de
v-rootcoord-dml_8_444129503507318440v1] [replicaID=444129521123196939] [taskType=Query] [nodeID=19] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [delegator/delegator.go:471] ["Delegator execute subTask failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel
=by-dev-rootcoord-dml_8_444129503507318440v1] [replicaID=444129521123196939] [taskType=Query] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [delegator/delegator.go:326] ["Delegator query failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel=by-dev-ro
otcoord-dml_8_444129503507318440v1] [replicaID=444129521123196939] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [querynodev2/handlers.go:187] ["failed to query on delegator"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [msgID=444142320382377985] [collectionID=4441
29503507318440] [channel=by-dev-rootcoord-dml_8_444129503507318440v1] [scope=All] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [segments/segment.go:488] ["validate fields failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [partitionID=44412950
3507318441] [segmentID=444129503507518683] [error="vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [querynodev2/services.go:919] ["failed to query channel"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [msgID=444142320382377985] [collectionID=444129503
507318440] [channel=by-dev-rootcoord-dml_7_444129503507318440v0] [scope=Historical] [error="vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [delegator/delegator.go:451] ["failed to execute sub task"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel=by-de
v-rootcoord-dml_7_444129503507318440v0] [replicaID=444129521123196939] [taskType=Query] [nodeID=19] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [delegator/delegator.go:471] ["Delegator execute subTask failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel
=by-dev-rootcoord-dml_7_444129503507318440v0] [replicaID=444129521123196939] [taskType=Query] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [delegator/delegator.go:326] ["Delegator query failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [channel=by-dev-ro
otcoord-dml_7_444129503507318440v0] [replicaID=444129521123196939] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [querynodev2/handlers.go:187] ["failed to query on delegator"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [msgID=444142320382377985] [collectionID=4441
29503507318440] [channel=by-dev-rootcoord-dml_7_444129503507318440v0] [scope=All] [error="worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [proxy/task_query.go:493] ["QueryNode query result error"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collection=444129503507318440] [partitionIDs="[]
"] [nodeID=19] [channels="[by-dev-rootcoord-dml_7_444129503507318440v0]"] [errorCode=UnexpectedError] [reason="failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not
 allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [proxy/lb_policy.go:176] ["search/query channel failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [collectionName=b
ook] [channelName=by-dev-rootcoord-dml_7_444129503507318440v0] [nodeID=19] [error="fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT
 index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] ["retry time"=0] [error="failed to search/query delegator 19 for 
channel by-dev-rootcoord-dml_7_444129503507318440v0: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"] [error
Verbose="failed to search/query delegator 19 for channel by-dev-rootcoord-dml_7_444129503507318440v0: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output 
fields for IVF_FLAT index is not allowed\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/go/src/github.com/milvus-io/mi
lvus/internal/proxy/lb_policy.go:182\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | github.com/milvus-io/milvus/internal/proxy.(*LBP
olicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/go/src/github.com/milv
us-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/run
time/asm_amd64.s:1598\nWraps: (2) failed to search/query delegator 19 for channel by-dev-rootcoord-dml_7_444129503507318440v0\nWraps: (3) fail to Query, QueryNode ID = 19, reason=failed to query channel, err=wor
ker(19) query failed: vector output fields for IVF_FLAT index is not allowed\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"] [stack="github.com/milvus-io/milvus/pkg/util/
retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_po
licy.go:153\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/p
kg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [proxy/task_query.go:493] ["QueryNode query result error"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collection=444129503507318440] [partitionIDs="[]
"] [nodeID=19] [channels="[by-dev-rootcoord-dml_8_444129503507318440v1]"] [errorCode=UnexpectedError] [reason="failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not
 allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [proxy/lb_policy.go:176] ["search/query channel failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [collectionName=b
ook] [channelName=by-dev-rootcoord-dml_8_444129503507318440v1] [nodeID=19] [error="fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT
 index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [WARN] [proxy/lb_policy.go:176] ["search/query channel failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [collectionName=b
ook] [channelName=by-dev-rootcoord-dml_8_444129503507318440v1] [nodeID=19] [error="fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT
 index is not allowed"]
milvus-standalone | [2023/09/09 14:08:15.463 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] ["retry time"=0] [error="failed to search/query delegator 19 for 
channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed"] [error
Verbose="failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output 
fields for IVF_FLAT index is not allowed\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/go/src/github.com/milvus-io/mi
lvus/internal/proxy/lb_policy.go:182\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | github.com/milvus-io/milvus/internal/proxy.(*LBP
olicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/go/src/github.com/milv
us-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/run
time/asm_amd64.s:1598\nWraps: (2) failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1\nWraps: (3) fail to Query, QueryNode ID = 19, reason=failed to query channel, err=wor
ker(19) query failed: vector output fields for IVF_FLAT index is not allowed\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"] [stack="github.com/milvus-io/milvus/pkg/util/
retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_po
licy.go:153\ngithub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/p
kg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"]
milvus-standalone | [2023/09/09 14:08:15.664 +00:00] [INFO] [proxy/meta_cache.go:875] ["clearing shard cache for collection"] [collectionName=book]
milvus-standalone | [2023/09/09 14:08:15.664 +00:00] [INFO] [proxy/meta_cache.go:875] ["clearing shard cache for collection"] [collectionName=book]
milvus-standalone | [2023/09/09 14:08:15.664 +00:00] [INFO] [querycoordv2/services.go:840] ["get shard leaders request received"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:15.665 +00:00] [INFO] [querycoordv2/services.go:840] ["get shard leaders request received"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440]
milvus-standalone | [2023/09/09 14:08:15.665 +00:00] [WARN] [proxy/lb_policy.go:126] ["no available shard delegator found"] [collectionID=444129503507318440] [collectionName=book] [channelName=by-dev-rootcoord-d
ml_8_444129503507318440v1] [nodes="[19]"] [excluded="[19]"]
milvus-standalone | [2023/09/09 14:08:15.665 +00:00] [WARN] [proxy/lb_policy.go:156] ["failed to select node for shard"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [collectionNa
me=book] [channelName=by-dev-rootcoord-dml_8_444129503507318440v1] [nodeID=-1] [error="no available shard delegator found: service unavailable"] [errorVerbose="no available shard delegator found: service unavail
able\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrServiceUnavailable\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:161\n  | github.com/mi
lvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:129\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetr
y.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | gi
thub.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl
).Execute.func2\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n 
 | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) no available shard delegator found\nWraps: (3) service unavailable\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (
3) merr.milvusError"]
milvus-standalone | [2023/09/09 14:08:15.665 +00:00] [WARN] [proxy/lb_policy.go:126] ["no available shard delegator found"] [collectionID=444129503507318440] [collectionName=book] [channelName=by-dev-rootcoord-d
ml_7_444129503507318440v0] [nodes="[19]"] [excluded="[19]"]
milvus-standalone | [2023/09/09 14:08:15.665 +00:00] [WARN] [proxy/lb_policy.go:156] ["failed to select node for shard"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collectionID=444129503507318440] [collectionNa
me=book] [channelName=by-dev-rootcoord-dml_7_444129503507318440v0] [nodeID=-1] [error="no available shard delegator found: service unavailable"] [errorVerbose="no available shard delegator found: service unavailable\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrServiceUnavailable\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:161\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:129\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) no available shard delegator found\nWraps: (3) service unavailable\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone | [2023/09/09 14:08:15.841 +00:00] [INFO] [observers/collection_observer.go:122] ["observes partitions timeout"] [partitionNum=1]
milvus-standalone | [2023/09/09 14:08:15.841 +00:00] [INFO] [observers/collection_observer.go:159] ["observe partitions status"] [partitionNum=1]
milvus-standalone | [2023/09/09 14:08:15.852 +00:00] [INFO] [balance/rowcount_based_balancer.go:205] ["balance channel"] ["online nodes"="[19]"] ["offline nodes"="[]"]
milvus-standalone | [2023/09/09 14:08:16.065 +00:00] [WARN] [proxy/task_query.go:416] ["fail to execute query"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [collection=444129503507318440] [partitionIDs="[]"] [requestType=query] [error="attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable"]
milvus-standalone | [2023/09/09 14:08:16.065 +00:00] [WARN] [proxy/task_scheduler.go:460] ["Failed to execute task: "] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [error="failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable"] [errorVerbose="failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\nWraps: (2) failed to query\nWraps: (3) attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed\nWraps: (4) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:45\n  | [...repeated from below...]\nWraps: (5) attempt #1\nWraps: (6) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrServiceUnavailable\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:161\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:129\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (7) no available shard delegator found\nWraps: (8) service unavailable\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.withPrefix (8) merr.milvusError"]
milvus-standalone | [2023/09/09 14:08:16.066 +00:00] [WARN] [proxy/impl.go:2706] ["Query failed to WaitToFinish"] [traceID=f1a57e27567ab2cb77f02ba81b1260f5] [role=proxy] [db=trac_data] [collection=book] [partitions="[]"] [error="failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable"] [errorVerbose="failed to query: attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed: attempt #1: no available shard delegator found: service unavailable\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:417\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:457\nWraps: (2) failed to query\nWraps: (3) attempt #0: failed to search/query delegator 19 for channel by-dev-rootcoord-dml_8_444129503507318440v1: fail to Query, QueryNode ID = 19, reason=failed to query channel, err=worker(19) query failed: vector output fields for IVF_FLAT index is not allowed\nWraps: (4) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:45\n  | [...repeated from below...]\nWraps: (5) attempt #1\nWraps: (6) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrServiceUnavailable\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:161\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:129\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:154\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:153\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:206\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (7) no available shard delegator found\nWraps: (8) service unavailable\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.multiErrors (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.withPrefix (8) merr.milvusError"]

### Anything else?

_No response_
stevetracvc commented 1 year ago

I decided to try this with v2.2, and search appears to work fine. So this does seem like a bug in 2.3

xiaofan-luan commented 1 year ago

for cosine metrics this is not supported. We will support it soon

xiaofan-luan commented 1 year ago

/assgin @liliu-z

stevetracvc commented 1 year ago

@xiaofan-luan oh wow, yes I think dropping the COSINE index and adding an IP index worked.

Is there any particular reason why? Anything I can do to help? I'm new to this software so I don't know the codebase at all, but if you have an idea of where I should look, I might be able to help.

xiaofan-luan commented 1 year ago

@xiaofan-luan oh wow, yes I think dropping the COSINE index and adding an IP index worked.

Is there any particular reason why? Anything I can do to help? I'm new to this software so I don't know the codebase at all, but if you have an idea of where I should look, I might be able to help.

This is due to the implementation. We are targeting at support this feature soon in 2.3.2

yanliang567 commented 1 year ago

close for now as comments above

xiaofan-luan commented 1 year ago

keep it until done

liliu-z commented 1 year ago

/assign

chathushka1996 commented 1 year ago

Is this fixed in version 2.3.2 npm pacakge?

stevetracvc commented 1 year ago

@chathushka1996 I thought they're only on a server release version of 2.3.0.

Either way, if your vectors are normalized, IP and cosine are the same thing. Inner Product is just the dot product, which is ||A|| ||B|| cos(theta). If your vectors are normalized then ||A|| == ||B|| == 1 (or very very close with floating point errors).

IP indexes work, cosine indexes don't. So if you can normalize your vectors (if they aren't already), then you can drop the cosine index and replace it with an IP index and everything should work fine.

liliu-z commented 1 year ago

@chathushka1996 Yes, this will be in 2.3.2

And thanks @stevetracvc for this explanation. After normalization, you can feel free to use either IP or L2 as distance to get correct Cosine topK. One more thing need to be careful is that normed-IP distance is exactly Cosine distance form numeric perspective but L2 is not. So IP is recommended to bypass this for now.

luancaarvalho commented 11 months ago

Does we have any mention to this limitation in the docs ? I didn’t found anything about it

xiaofan-luan commented 11 months ago

Does we have any mention to this limitation in the docs ? I didn’t found anything about it

this is mentioned in knowhere release note. Again this shouldn't be happend and we already worked on fixing it

stale[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

Alan-2018 commented 5 months ago

mark

yanliang567 commented 5 months ago

this was already supported, please try Milvus 2.3.12