search

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
143 stars 108 forks source link

[BUG] KNN doesn't release memory when close index #1012

Open ylwu-amzn opened 11 months ago

ylwu-amzn commented 11 months ago

Run big KNN indexes, and see memory usage keep high as 95%. Close one big KNN index, but memory usage still keep high. Have to delete the KNN index to release the memory. After deleting the index, can see memory usage goes down to 75%.

Seems like a bug that when close an index, KNN doesn't release the memory.

jmazanec15 commented 11 months ago

@ylwu-amzn Thanks for reporting this. Yes, the problem is that we do watch for the files to be deleted to free memory as opposed to freeing memory when the IndexReader gets closed.

I think a fix for this is to implement a DocValuesReader that frees memory when it is closed.