[BUG]k-nn approximate search does not work after reindex

[mshahmoradi87]

What is the bug? we had an index with 5 shards and we wanted to rindex it to a new index with one shard. approximate search does not show results with exact same vector in the new index.

Happens in both 1.3.0.0 and 2.5.0.0

How can one reproduce the bug? Here are the steps we took:

1- we have created new index with exact same mappings and the only different is number of shards and replicas 2- we did full reindex from source index to new index 3- wait untill reindex is finished and new index showing same count number of documents 4- we did a search for a document with to make sure it is returned as a result 5- we did knn query and that specific document that has exact same vector with the one that exists in the new index is not retruned. It shows some result with low score even there are records with same match in the index. 6- we do same search in old index and that document is in the results with score 1.

knn_vector definiton in mapping -->

"imageVector": { "type": "knn_vector", "dimension": 256, "method": { "name": "hnsw", "space_type": "l2", "engine": "faiss", "parameters": { "ef_search": 16, "ef_construction": 512, "m": 32 }

search query: GET knn_new_index/_search { "_source": ["field1", "field2", "field3", "field4"], "query": { "knn": { "imageVector": { "vector": [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ], "k": 50 } } }, "size": 50 }

What is the expected behavior? The expectation is to get most similar results for search queries after reindex.

I have tried _refresh _forcemerge _update_by_query _flush on the new index and sometimes it fixes the issue and some time it does not. As it has random behaviour, I could not proceed to production environment.

Is there any way to force knn to rebuild scores and graphs based on the data in the index ?

[navneet1v]

@mshahmoradi87 can you query the index mapping of the new index and validate that new index has the right mappings?

[martin-gaievski]

any update on this @mshahmoradi87? if issue isn't active we may need to close it

[mshahmoradi87]

@mshahmoradi87 can you query the index mapping of the new index and validate that new index has the right mappings?

I did, and mappings are correct and exactly the same. I have tried to reproduce the issue with some sample data, but it does not happen with small dataset. But it appears on our original indexes. for now, the workaround was to create new indexes and wait until the current index is expired from retention perspective.

[heemin32]

@mshahmoradi87 This is expected behavior as you reduced the number of shard from 5 to 1.

In general, having more shard gives better recall because you are actually searching k * 5(shard number) neighbors when everything else is same.

I recommend to increase ef_search value to 100 or at least greater than k value. Then, you will get better recall with even single shard.

[jmazanec15]

@mshahmoradi87 are you still running into any issues here?