opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
152 stars 113 forks source link

Generalize lib interface to return context objects #1925

Closed jmazanec15 closed 1 month ago

jmazanec15 commented 1 month ago

Description

Generalizes the KNNLibrary interface to return an object for both search and indexing so that the plugin can search/index against them. The names of the methods are "getKNNLibraryIndexBuildContext" and "getKNNLibrarySearchContext". In the future, I could see these methods changing into something like getKNNLibraryIndexBuilder and getKNNLibrarySearcher so that the engine actually encapsulates the interactions with the libraries, but that needs some more thought. For now, this change makes it more clear what one of the central roles of the KNNLibrary interface is - telling the plugin how to index/search.

For the quantization framework, making the interface return objects will allow us to properly configure both quantization as well as whatever index we want to use with it. See the following locations for more details where we would do this:

  1. https://github.com/jmazanec15/k-NN-1/blob/method-stage/src/main/java/org/opensearch/knn/index/mapper/MethodFieldMapper.java#L63
  2. https://github.com/jmazanec15/k-NN-1/blob/method-stage/src/main/java/org/opensearch/knn/index/query/KNNQueryBuilder.java#L376

Again, no functionality change - just more refactoring to better support the quantization framework.

One thing I have realized from these changes is that he KNNMethodContext class needs to be sured-up a bit to be more robust. I will create a separate issue for this.

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

heemin32 commented 1 month ago

How about getKNNLibraryIndexingContext?

jmazanec15 commented 1 month ago

Addressed @heemin32 !