[Feature]: Replace Milvus scalar execution engine with velox - Githubissues

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications

https://milvus.io

Apache License 2.0

31.12k stars 2.95k forks source link

[Feature]: Replace Milvus scalar execution engine with velox #21141

Closed xiaofan-luan closed 1 month ago

xiaofan-luan commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

implement more datatypes, such as map, list, geo spatial
implement more operators, sucha as aggeration https://github.com/milvus-io/milvus/issues/19152 wild card matching. https://github.com/milvus-io/milvus/issues/19913
do scalar search faster -> fully utilized vectorization on columnar data.
for the preparation of further of SQL support

Describe the solution you'd like.

We are investigating on the possibility to use https://github.com/facebookincubator/velox, which is execution engine opensourced by Meta.

[ ] investigate combing milvus storage format with velox source @sunby
[ ] compare performance of velox and duckdb @zhagnlu
[ ] implement a straightforward demo with velox to support filter expression @zhagnlu
[ ] think of the plan to integrate velox together with Milvus SQL Query

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

xiaofan-luan commented 1 year ago

@zhagnlu pls update some progress on this feature

xiaofan-luan commented 1 month ago

this is most likely done by customize some part of velox into milvus. close it for now