[Discussion] What types of k-NN documentation should go into search documentation section?

[jmazanec15]

Description

For k-NN, we have the bulk of our documentation in the path "search-plugins/knn/index/" (ref). For instance we have our scripting extensions, information about our field mapping (https://opensearch.org/docs/2.13/search-plugins/knn/knn-index/), our API documentation, settings, filter based search tutorials, etc. I think from the users perspective, this may be hard to navigate. Also, from dev perspective, its not clear what should go where.

I wanted to see what others thought about what portions of the k-NN documentation should go in this section.

IMO I think it makes sense for this section to focus on walk through/tutorials for the different types of search systems that can be built. So, for instance, the k-NN search section may have the following sub-sections:

1. Getting started with k-NN in OpenSearch
2. Performance tuning for OpenSearch k-NN
3. Choosing the right k-NN configuration for OpenSearch
4. k-NN search with filters
5. k-NN search with nested fields

Then, docs for specific functionality get moved to the more general technical sections of documentation:

1. APIs -> https://opensearch.org/docs/latest/api-reference/
2. FieldType -> https://opensearch.org/docs/latest/field-types/
3. Queries -> https://opensearch.org/docs/latest/query-dsl/
4. Script score -> not sure if implemented yet
5. Painless extensions -> not sure if implemented yet

cc: @navneet1v @vamshin @dylan-tong-aws

[hdhalter]

I like the task-based approach that you have suggested. Most often, users read documentation to perform a task, so we want to enable them to complete the task as quick as possible. Also interested in others' opinions.

[navneet1v]

I also like the idea of task based, but I would like to put those tasks(which can be getting started with OS vector search, some example flows etc) here : https://opensearch.org/platform/search/vector-database.html rather than in main documentation.

In Main documentation we should think of a better way to provide details. Things which are little technical with example can be put in main documentation.

@hdhalter I want to know 1 more thing as we are thinking about this documentation, what would be the right place to put the architecture aka internal functioning of plugin. We get some questions around that too.

[hdhalter]

The opensearch.org pages other than the docs pages (such as the Platform pages) tend to be marketing-focused, describing what functionality exists in OpenSearch. The details of how to use OpenSearch should be in the documentation. Even blogs should separate the what from the how except when describing specific use cases. So, getting started and performance tuning should be in the docs. The marketing pages should point to the docs, as applicable.

We know that the docs tend to be more reference-oriented, but would like to move toward being more task-oriented.

As far as plugin architecture, that is typically in the plugin repo.

[jhmcintyre]

+1 to Heather's comment; the vector database capabilities page is more of a high-level, marketing-oriented overview. What makes sense to me would be comprehensive guidance to getting started with vector functionality hosted in documentation when that content is ready, which I would then link to in the "getting started" section of the capabilities page.

[jmazanec15]

@hdhalter this makes sense. This may be a silly question, but if we were to make substantial changes to the doc structure for k-NN, is there any backwards compatibility concern? For example, the k-NN docs are completely different in 2.14 than they are in 2.13. I assume no, but wanted to check.

[hdhalter]

@hdhalter this makes sense. This may be a silly question, but if we were to make substantial changes to the doc structure for k-NN, is there any backwards compatibility concern? For example, the k-NN docs are completely different in 2.14 than they are in 2.13. I assume no, but wanted to check.

It's not a silly question, as the way we manage versions may not be widely known. We've taken a policy to only support the latest minor version of a major release. So any changes we make now, for example, would only be backported to 2.13 (and 1.3 in most cases), and continue forward. If we decided to make changes to this section for 2.14, the changes would be made in 2.14 and forward. We try to point people to the latest version if they are in an 'archived' version so they are seeing the most up to date. We indicate when a feature is released in case they are working in a previous release. Maybe that's more info than you wanted, but hope it makes sense!