Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data

https://opensearch.org/docs/latest/search-plugins/knn/index/

Apache License 2.0

152 stars 113 forks source link

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

Closed shatejas closed 1 month ago

shatejas commented 1 month ago

Description

The commit merges feature/iterative-index-build branch
It integrates with KNNVectorValues and NativeEngines990KnnVectorsFormat
There are changes to reuse offheap vector buffer during iterative vector transfer

Related Issues

Resolves #1853

Check List

[x] New functionality includes testing.
[x] New functionality has been documented.
[ ] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v commented 1 month ago

@shatejas lets fix the CIs first

navneet1v commented 1 month ago

Some of the comments can be old as there were updates in the PR. Please ignore comments if they are already answered.

navneet1v commented 1 month ago

Overall code looks good to me. CIs are failing due to some backward incompatible changes in main branch of Opensearch. That needs to be fixed first and then CIs needs to be run again. Approving this PR assuming CIs will pass. But lets not merge the PR till CIs are passing.

opensearch-trigger-bot[bot] commented 1 month ago

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1950-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fd59b9adf42b07aa2b2058c4badff6dacf8306a8
# Push it to GitHub
git push --set-upstream origin backport/backport-1950-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1950-to-2.x.