Optimizes live docs computes for force merge in NativeEngines990KNNVectorWriter

shatejas commented 2 months ago

Description

Currently live docs are computed with linear complexity even when it does not have deleted docs. This PR skips computing it

Testing

UTs and javadocs in progression. Need initial feed back

Check List

[x] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v commented 2 months ago

Please also update the changelog.

shatejas commented 1 month ago

After more experiments we figured that this live docs iteration is not the root cause, This only yields 32ms gains for 1.6M vectors adding an overhead of maintaining the merge vectors code.

Decided to discard this

opensearch-project / k-NN