search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.78k stars 2.86k forks source link

[Feature]: Error when creating collection: schema does not contain vector field: invalid parameter #33853

Open lynnssi opened 3 months ago

lynnssi commented 3 months ago

Is there an existing issue for this?

Is your feature request related to a problem? Please describe.

In Milvus 2.4.1, I am able to create collections that do not have a vector field in the schema. When I upgraded to 2.4.4, I encountered the error RPC error: [create_collection], <MilvusException: (code=1100, message=schema does not contain vector field: invalid parameter)>.

I would like to be able to use this functionality of vector-less collections in future versions of Milvus

Example code

from pymilvus import MilvusClient
client_241 = MilvusClient(URL241)
client_244 = MilvusClient(URL244)

from pymilvus import DataType
schema = MilvusClient.create_schema(auto_id=False, enable_dynamic_field=True)
schema.add_field(field_name="key", datatype=DataType.VARCHAR, is_primary=True, max_length=20)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, is_primary=False, max_length=1000)
schema.add_field(field_name="value", datatype=DataType.INT64, is_primary=False)

# Creates collection without error
client_241.create_collection("test_no_vector_field2", schema=schema)

# Fails to create collection
client_244.create_collection("test_no_vector_field", schema=schema)
>> RPC error: [create_collection], <MilvusException: (code=1100, message=schema does not contain vector field: invalid parameter)>

Describe the solution you'd like.

I would like to be able to create, insert into, and query collections that do not have a vector field in future versions of Milvus. This is very useful for the development of search applications.

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

sunby commented 3 months ago

Hi @lynnssi , may I ask that in what scenario would you need to use a vector database without requiring a vector field? Milvus is a vector database and it's python sdk always check the vector field. So we add this check in milvus server side recently.

junjie-landing commented 2 months ago

I have the same issue. Reason is simple, I have duplicate metadata that I do not want to flatten/denormalized into the table, which makes it very redundant.

(It's bit like relational db. I know milvus is not meant for that, but I wouldn't want to put meta data in yet another RDBMS, which simplified increase our tech stack complexity.)

BTW, this is a breaking change to our techstack. To avoid reimplementing the workflow, I'm gonna add a dummy vector field and bypass for now.

xiaofan-luan commented 2 months ago

Milvus is not designed for this. Even though milvus will support "data in data out " feature soon, where vector field is not required any more, we still expect data to be embedded into a vector field.

Also, Milvus won't support any operation like Join so store meta into multiple collection should not be a solution