Feature: Merging Small Indexes

unum-cloud / usearch

Fast Open-Source Search & Clustering engine × for Vectors & 🔜 Strings × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍

https://unum-cloud.github.io/usearch/

Apache License 2.0

2.12k stars 126 forks source link

Feature: Merging Small Indexes #84

Open ashvardanian opened 1 year ago

ashvardanian commented 1 year ago

Describe what you are looking for

As a performance optimization, we can implement merge for existing indexes instead of moving (inserting) elements one by one.

Can you contribute to the implementation?

[X] I can contribute

Is your feature request specific to a certain interface?

It applies to everything

Contact Details

No response

Is there an existing issue for this?

[X] I have searched the existing issues

Code of Conduct

[X] I agree to follow this project's Code of Conduct

rationalga commented 6 months ago

Hi, Has this feature been developed or not yet? While loading indexes that are bigger than RAM is great but its not useful if we can't create them in the first place i.e. during insertion index size can't outgrow the RAM (as far as I know). Thank you very much.

mbautin commented 3 weeks ago

It would also be great to allow filtering entries during this merge process based on a predicate (e.g. filtering out entries that were later marked as deleted).