opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
156 stars 123 forks source link

[Backport 2.x] Preloads .vec and .vex files #2189

Closed shatejas closed 1 month ago

shatejas commented 1 month ago

LuceneFlatVectorReader uses IOContext.Random to open the read. IOContext.Random indicates the kernel to not read ahead the pages on to physical memory. This causes an increase in merge time due to increase of read ops at runtime.

The preload settings signals the kernal to preload the files when the reader is opened

Description

Backport for https://github.com/opensearch-project/k-NN/pull/2186/files

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v commented 1 month ago

@shatejas does additional settings didn’t work and we have to use additional index settings?