refactor-improve-indexers - Githubissues

jina-ai / executors

internal-only

Apache License 2.0

31 stars 12 forks source link

refactor-improve-indexers #292

Closed numb3r3 closed 2 years ago

numb3r3 commented 2 years ago

Changes:

use limit, deprecate top_k
use add_with_ids and remove_ids to improve update and delete implement
support ef_query and ef_construct parameter for HNSW index
add /status and /clear endpoints into FaissSearcher
use index_traversal_paths and search_traversal_paths as parameters
refactor online /train endpoint
- use PSQL to store trained index
- fetch the stored index from PSQL at initialization step

TODO:

[x] integrate test for online & offline training. refer to https://github.com/jina-ai/executors/blob/20a86270c2705b83ebdb2b14cf3809ed1019ee06/tests/integration/faiss_psql/test_train_psql.py

cristianmtr commented 2 years ago

We also need more thorough testing of these new features:

[x] integration test with separate Flows (one for Index + Search; one for Training)
- [x] with different Faiss index keys
[x] training / storing / retrieving the model