[FEATURE] Add high level DSL for k-nn plugin

[moshfrid]

As the ANN plugin is out, it would be great to have an index and field for it

[wbeckler]

Would you be up for creating the PR?

[VachaShah]

Transferring this issue here as opensearch-dsl-py is being merged into opensearch-py as part of https://github.com/opensearch-project/opensearch-py/issues/194

[wbeckler]

I believe https://github.com/opensearch-project/opensearch-py/issues/291 is a related request.

[lucafrost]

I believe #291 is a related request.

pinging @armenabnousi as they may be interested...

I'm down to create a PR to introduce k-NN APIs into opensearch-py. It would be tremendously helpful to get some community feedback on what features and abstractions people want to see. There's clearly demand for (k-NN) plugin integration cf. #177 #291 etc.

Per @arnavdas88's suggestion, I think it would make sense to incorporate k-NN via client.knn as below...

from opensearchpy import OpenSearch

client = OpenSearch(...)

my_vector = [1, 2, 3, ... 1024]

# approx k-NN
client.knn.approx(index="example", field="vector", vector=my_vector, size=10)

# exact k-NN
query = {
    "query": {
        "match_all": {}
    }
}
client.knn.exact(index="example", body=query, field="vector", vector=my_vector, size=10)

This would be a nice augmentation to existing search functionality for k-NN users. What APIs from k-NN plugin API would you like to see?

Perhaps methods like GET /_plugins/_knn/models/{model_id} could become client.knn.models.get(), .delete(), .train(), and so on...

[dblock]

One can do k-nn in the client today, see https://github.com/opensearch-project/opensearch-py/blob/main/guides/plugins/knn.md, renamed this issue to add a higher level DSL.

[dblock]

A mechanism for plugins was added in https://github.com/opensearch-project/opensearch-py/pull/93. Hopefully someone can pick this one up.

[saimedhi]

I will work on adding specifications for the knn-plugin and fine-tuning the generator for plugin generation.

[saimedhi]

I am closing this issue since KNN APIs are now available in opensearch-py. If you feel the issue is not resolved, please feel free to reopen it.

[kenmasumitsu]

I can see docs at https://github.com/opensearch-project/opensearch-py/blob/main/guides/plugins/knn.md. But it is low level API.

Are there high level DSL API for k-nn plugin?

[dblock]

Are there high level DSL API for k-nn plugin?

Do you have an example of what we have vs. what you'd like to write? Maybe we should reopen this.

[kenmasumitsu]

This page explains High level Python API. But no description on knn vector.

I want to know

• how to declare field of knn_vector type.
• how to search with knn_vector.

[dblock]

There's no higher level DSL implemented indeed. You should open a feature request for it, but that's basically my question: what would you like to write in Python that doesn't look like https://github.com/opensearch-project/opensearch-py/blob/main/samples/knn/knn_basics.py#L78. Include that in that issue. Note there are some related issues like https://github.com/opensearch-project/opensearch-py/issues/630 too.

[kenmasumitsu]

If there is a High level API, it would be natural to try that first.

If the High level API does not support some features, it would be kind to describe it in the documentation. It would be unfortunate to implement a new feature using the High level API, only to find out halfway through that it is not supported and have to re-do it.

[dblock]

I agree @kenmasumitsu. Help us out by contributing?