search

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
152 stars 113 forks source link

Add DocValuesProducers for releasing memory when close index #1946

Closed luyuncheng closed 2 weeks ago

luyuncheng commented 1 month ago

Description

in #1885 we talk about a method that we need release memory when a producer closed. so i open this PR and added KNN80DocValuesProducer. this producer can release memory when reader closed a segment.

also i added refCount as the comments cares about.

i think this pr is the 1st step, we only added a producers, because we talked in #1885, that we need get binaryDocValues in DocValuesProducers from native engine.

i see there is an #1853 on going, so i prefer to in next step to read binaryDocValues from native engines.

Related Issues

https://github.com/opensearch-project/k-NN/issues/1885

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v commented 1 month ago

@luyuncheng can we fix the build CIs. BWC CIs is failing across other PRs will check whats happening there.

kotwanikunal commented 1 month ago

Added some thoughts related to the cache free up: https://github.com/opensearch-project/k-NN/issues/1885#issuecomment-2284961955

luyuncheng commented 1 month ago

Can we check if FilterDirectory unwraps to FSDirectory before casting? If it does not, can we warn log and NO-OP? My main concern here is around remove directory implementations.

@jmazanec15 at https://github.com/opensearch-project/k-NN/pull/1946/commits/2bcc13946a5c26a25f0183321ecdba876dc1d40a how about added try catch ClassCastException

navneet1v commented 3 weeks ago

@luyuncheng can you please look at the comment added reply on them.

luyuncheng commented 3 weeks ago

@luyuncheng can you please look at the comment added reply on them.

@navneet1v Sorry for the late reply, i just get back from vacation. i'll do it ASAP.

luyuncheng commented 2 weeks ago

@jmazanec15 @navneet1v Thanks guys for reviewing