[Feature]: Add Sparse Float Vector support

[zhengbuqian]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

Now milvus supports only dense vectors and lack the ability to store/index/search sparse vectors(vectors with up to million dimensions while only a handful of them are non zero). We wish to add sparse float vector support to Milvus so users can insert, index and search them with ease.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[zhengbuqian]

This comment is to track the implementation progress of sparse vector support in Milvus.

• Happy path of the general usage:
  • [x] Create a collection with a sparse vector type field
  • [x] Insert sparse vectors into the collection and persist to binlog
  • [x] Load sparse vectors from binlog for brute force search, segment compaction, etc
  • [x] Build index to accelerate searching on sparse vector field with tunable recall/performance
  • [x] The ability to perform index search
  • [x] The ability to retrieve the original vectors(GetVectorByIds)
  • [x] index config settings(for both growing and sealed index)
  • [x] Sealed segment to generate temp binlog index
  • [x] #30357
  • [x] #30629
  • [x] #30630
  • [ ] #31062
  • [x] Support various sparse input format in python SDK
  • [x] https://github.com/milvus-io/pymilvus/pull/1902
• Must Haves before GA
  • [ ] Hybrid search with MultiVector
  • [x] Milvus Client
  • [ ] Monitoring and Metrics
  • [x] Documentation and User Guide
  • [ ] Add sparse vector support in import utils, support pandas pd.SparseType
  • [ ] Add sparse vector support in restful httpserver
• Testing TODOs
  • [x] Fix/Add unit tests for all modified files in PR
  • [x] End 2 End tests
• Advanced feature support:
  • [ ] Range search and search iterator
  • [ ] Knowhere Iterator + GroupBy
  • [ ] ...
• Future
  • [ ] support Storage v2

Pending:

• [ ] Do we need sparse vector support in row based message?
• [ ] internal/core/src/storage/Util.cpp and parquet_c.h how is this used? How do we handle sparse?
• [ ] Mmap support for sparse field data

[zhengbuqian]

Knowhere tracking issue: https://github.com/zilliztech/knowhere/issues/193

[xiaofan-luan]

Exciting about the new feature!

[zhengbuqian]

basic sparse support has been added to master branch with the merge of #30357, #30629, #30630 and pymilvus #1920.

[zhengbuqian]

For SDK owners:

We also need to support sparse float vector in C#/NodeJs/Java/Go SDK.

• PyMilvus PR for reference: https://github.com/milvus-io/pymilvus/pull/1902
• Python code example to use sparse float vector: https://github.com/milvus-io/pymilvus/blob/master/examples/hello_sparse.py
• Data representation in proto: https://github.com/milvus-io/milvus-proto/blob/5423fd39943c646c722a08617402f4a890e40e74/proto/schema.proto#L131

The accepted sparse input format:

• if your language community already has a widely used representation of sparse float vector, we should accept that. For example PyMilvus supports the scipy.sparse representations as input.
• if not, support a map representation like {30: 0.34, 78: 0.11, 22: 0.66}.

When sending requests to milvus(both insert and search), use one proto bytes to represent a single sparse vector, and encode it as densely packed bytes: idx, val, idx, val, .... Indices in the packed bytes should be in uint32 range and ordered in ascending order(the user input can be unordered though). No duplicate indices allowed.

Note that support for those SDKs is not a must-have for the formal milvus 2.4 release. We'll be adding more features for sparse and announcing GA in the next major release(2.5 or 2.6). I'll keep updating the issues as necessary.

Thanks a lot for the efforts!